2020-06-04T08:08:57.815 INFO:root:teuthology version: 1.0.0-c66e34b 2020-06-04T08:08:57.818 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name yuvalif-2020-06-04_08:07:49-rgw:multisite-notifications_persistent_delivery2-distro-basic-smithi --description rgw:multisite/{clusters.yaml frontend/beast.yaml omap_limits.yaml overrides.yaml realms/three-zone-plus-pubsub.yaml tasks/test_multi.yaml valgrind.yaml} --block --archive /home/teuthworker/archive/yuvalif-2020-06-04_08:07:49-rgw:multisite-notifications_persistent_delivery2-distro-basic-smithi/5116651 --lock --owner scheduled_yuvalif@teuthology -- /tmp/teuthology-worker.chypc1t5.tmp 2020-06-04T08:08:57.858 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-06-04T08:08:57.998 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuvalif-2020-06-04_08:07:49-rgw:multisite-notifications_persistent_delivery2-distro-basic-smithi/5116651 branch: notifications_persistent_delivery2 description: rgw:multisite/{clusters.yaml frontend/beast.yaml omap_limits.yaml overrides.yaml realms/three-zone-plus-pubsub.yaml tasks/test_multi.yaml valgrind.yaml} email: null first_in_suite: false job_id: '5116651' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuvalif-2020-06-04_08:07:49-rgw:multisite-notifications_persistent_delivery2-distro-basic-smithi nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: notifications_persistent_delivery2 ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 setgroup: ceph setuser: ceph global: osd heartbeat grace: 40 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 osd fast shutdown: false rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) sha1: 5a141c6290e05d95ab4ea2df3c0cb561d6b8e984 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 5a141c6290e05d95ab4ea2df3c0cb561d6b8e984 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 workunit: branch: notifications_persistent_delivery2 sha1: 5a141c6290e05d95ab4ea2df3c0cb561d6b8e984 owner: scheduled_yuvalif@teuthology priority: 10 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 5a141c6290e05d95ab4ea2df3c0cb561d6b8e984 sleep_before_teardown: 0 suite: rgw:multisite suite_branch: notifications_persistent_delivery2 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_notifications_persistent_delivery2/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 5a141c6290e05d95ab4ea2df3c0cb561d6b8e984 tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.27832 2020-06-04T08:08:57.999 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_notifications_persistent_delivery2/qa; will attempt to use it 2020-06-04T08:08:58.015 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_notifications_persistent_delivery2/qa/tasks 2020-06-04T08:08:58.017 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-06-04T08:08:58.021 INFO:teuthology.task.internal:Checking packages... 2020-06-04T08:08:58.057 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '5a141c6290e05d95ab4ea2df3c0cb561d6b8e984' 2020-06-04T08:08:58.058 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-06-04T08:08:58.059 INFO:teuthology.packaging:ref: None 2020-06-04T08:08:58.059 INFO:teuthology.packaging:tag: None 2020-06-04T08:08:58.059 INFO:teuthology.packaging:branch: notifications_persistent_delivery2 2020-06-04T08:08:58.060 INFO:teuthology.packaging:sha1: 5a141c6290e05d95ab4ea2df3c0cb561d6b8e984 2020-06-04T08:08:58.060 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=notifications_persistent_delivery2 2020-06-04T08:08:58.265 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-2160-g5a141c6290e-1bionic 2020-06-04T08:08:58.266 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-06-04T08:08:58.278 INFO:teuthology.task.internal:no buildpackages task found 2020-06-04T08:08:58.278 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-06-04T08:08:58.294 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-06-04T08:08:58.295 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-06-04T08:08:58.355 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-06-04T08:09:08.369 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-06-04T08:09:18.390 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-06-04T08:09:28.409 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-06-04T08:09:38.422 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-06-04T08:09:48.442 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-06-04T08:09:58.463 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 0)... 2020-06-04T08:10:08.499 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-06-04T08:10:18.516 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 0 + 2, have 1)... 2020-06-04T08:10:28.539 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_yuvalif@teuthology', 'count': 2, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/yuvalif-2020-06-04_08:07:49-rgw:multisite-notifications_persistent_delivery2-distro-basic-smithi/5116651'} 2020-06-04T08:10:28.566 DEBUG:teuthology.lock.ops:locked ubuntu@smithi060.front.sepia.ceph.com, ubuntu@smithi196.front.sepia.ceph.com 2020-06-04T08:10:28.664 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f26b769da60>, signals=[15]) 2020-06-04T08:10:28.911 INFO:teuthology.provision.fog.smithi196:Scheduling deploy of ubuntu 18.04 2020-06-04T08:10:28.915 INFO:teuthology.provision.fog.smithi060:Scheduling deploy of ubuntu 18.04 2020-06-04T08:10:29.300 INFO:teuthology.orchestra.console:Power off smithi196 2020-06-04T08:10:29.303 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-06-04T08:10:29.332 INFO:teuthology.orchestra.console:Power off smithi060 2020-06-04T08:10:29.333 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-06-04T08:10:29.347 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-04T08:10:29.362 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-04T08:10:33.378 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-04T08:10:33.393 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-04T08:10:37.498 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-04T08:10:37.512 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-04T08:10:37.715 INFO:teuthology.orchestra.console:Power off for smithi196 completed 2020-06-04T08:10:37.728 INFO:teuthology.orchestra.console:Power off for smithi060 completed 2020-06-04T08:10:37.817 INFO:teuthology.orchestra.console:Power on smithi196 2020-06-04T08:10:37.817 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-06-04T08:10:37.839 INFO:teuthology.orchestra.console:Power on smithi060 2020-06-04T08:10:37.841 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-06-04T08:10:37.866 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-04T08:10:37.881 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-04T08:10:41.898 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-04T08:10:41.914 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-04T08:10:42.117 INFO:teuthology.orchestra.console:Power on for smithi196 completed 2020-06-04T08:10:42.218 INFO:teuthology.provision.fog.smithi196:Waiting for deploy to finish 2020-06-04T08:10:46.035 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-06-04T08:10:46.257 INFO:teuthology.orchestra.console:Power on for smithi060 completed 2020-06-04T08:10:46.358 INFO:teuthology.provision.fog.smithi060:Waiting for deploy to finish 2020-06-04T08:13:32.578 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:13:41.682 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:13:43.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:13:50.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:13:52.623 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:13:59.827 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:14:01.681 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:14:08.883 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:14:10.737 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:14:19.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:14:28.851 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:14:37.938 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:15:14.926 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:15:15.012 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-06-04T08:15:15.015 INFO:teuthology.orchestra.run.smithi060:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-06-04T08:15:15.371 INFO:teuthology.provision.fog.smithi060:Node is ready 2020-06-04T08:15:15.372 INFO:teuthology.orchestra.run.smithi060:> hostname 2020-06-04T08:15:15.420 INFO:teuthology.orchestra.run.smithi060.stdout:smithi060.front.sepia.ceph.com 2020-06-04T08:15:15.421 INFO:teuthology.orchestra.run.smithi060:> grep smithi060.front.sepia.ceph.com /etc/hosts 2020-06-04T08:15:15.469 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60 smithi060.front.sepia.ceph.com smithi060 2020-06-04T08:15:15.469 INFO:teuthology.orchestra.run.smithi060:> sudo hostname smithi060 2020-06-04T08:15:15.540 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:15:15.557 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-06-04T08:15:15.598 INFO:teuthology.orchestra.run.smithi060:> sudo sed -i -e 's/smithi060.front.sepia.ceph.com/smithi060/g' /etc/hosts 2020-06-04T08:15:15.617 INFO:teuthology.orchestra.run.smithi060:> sudo sed -i -e 's/172.21.15.60/172.21.15.60/g' /etc/hosts 2020-06-04T08:15:15.672 INFO:teuthology.orchestra.run.smithi060:> sudo sed -i -e 's/smithi060.front.sepia.ceph.com/smithi060/g' /etc/hostname 2020-06-04T08:15:15.730 INFO:teuthology.orchestra.run.smithi060:> sudo hostname smithi060 2020-06-04T08:15:15.783 INFO:teuthology.provision.fog.smithi060:Deploy complete! 2020-06-04T08:15:15.785 DEBUG:teuthology.parallel:result is None 2020-06-04T08:15:21.569 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:15:21.581 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-06-04T08:15:27.586 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:15:27.598 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-06-04T08:15:33.610 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:15:39.624 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:15:45.642 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:15:45.739 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-06-04T08:15:45.740 INFO:teuthology.orchestra.run.smithi196:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-06-04T08:15:46.075 INFO:teuthology.provision.fog.smithi196:Node is ready 2020-06-04T08:15:46.077 INFO:teuthology.orchestra.run.smithi196:> hostname 2020-06-04T08:15:46.128 INFO:teuthology.orchestra.run.smithi196.stdout:smithi196.front.sepia.ceph.com 2020-06-04T08:15:46.129 INFO:teuthology.orchestra.run.smithi196:> grep smithi196.front.sepia.ceph.com /etc/hosts 2020-06-04T08:15:46.175 INFO:teuthology.orchestra.run.smithi196.stdout:172.21.15.196 smithi196.front.sepia.ceph.com smithi196 2020-06-04T08:15:46.176 INFO:teuthology.orchestra.run.smithi196:> sudo hostname smithi196 2020-06-04T08:15:46.272 INFO:teuthology.orchestra.run.smithi196:> sudo sed -i -e 's/smithi196.front.sepia.ceph.com/smithi196/g' /etc/hosts 2020-06-04T08:15:46.288 INFO:teuthology.orchestra.run.smithi196:> sudo sed -i -e 's/172.21.15.196/172.21.15.196/g' /etc/hosts 2020-06-04T08:15:46.346 INFO:teuthology.orchestra.run.smithi196:> sudo sed -i -e 's/smithi196.front.sepia.ceph.com/smithi196/g' /etc/hostname 2020-06-04T08:15:46.406 INFO:teuthology.orchestra.run.smithi196:> sudo hostname smithi196 2020-06-04T08:15:46.466 INFO:teuthology.provision.fog.smithi196:Deploy complete! 2020-06-04T08:15:46.467 DEBUG:teuthology.parallel:result is None 2020-06-04T08:15:46.612 INFO:teuthology.lock.ops:Checking smithi060.front.sepia.ceph.com 2020-06-04T08:15:46.613 INFO:teuthology.lock.ops:New key found. Updating... 2020-06-04T08:15:46.641 INFO:teuthology.lock.ops:Checking smithi196.front.sepia.ceph.com 2020-06-04T08:15:46.642 INFO:teuthology.lock.ops:New key found. Updating... 2020-06-04T08:15:46.663 INFO:teuthology.lock.ops:Updating [smithi060.front.sepia.ceph.com]: set os type and version on server 2020-06-04T08:15:46.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-04T08:15:46.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:15:46.790 INFO:teuthology.orchestra.run.smithi060:> true 2020-06-04T08:15:47.048 INFO:teuthology.orchestra.run.smithi060:> uname -m 2020-06-04T08:15:47.095 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2020-06-04T08:15:47.096 INFO:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2020-06-04T08:15:47.146 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="Ubuntu" 2020-06-04T08:15:47.147 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="18.04.4 LTS (Bionic Beaver)" 2020-06-04T08:15:47.147 INFO:teuthology.orchestra.run.smithi060.stdout:ID=ubuntu 2020-06-04T08:15:47.147 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE=debian 2020-06-04T08:15:47.147 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="Ubuntu 18.04.4 LTS" 2020-06-04T08:15:47.147 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="18.04" 2020-06-04T08:15:47.148 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://www.ubuntu.com/" 2020-06-04T08:15:47.148 INFO:teuthology.orchestra.run.smithi060.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-06-04T08:15:47.148 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-06-04T08:15:47.148 INFO:teuthology.orchestra.run.smithi060.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-06-04T08:15:47.148 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_CODENAME=bionic 2020-06-04T08:15:47.149 INFO:teuthology.orchestra.run.smithi060.stdout:UBUNTU_CODENAME=bionic 2020-06-04T08:15:47.150 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2020-06-04T08:15:47.172 INFO:teuthology.lock.ops:Updating [smithi196.front.sepia.ceph.com]: set os type and version on server 2020-06-04T08:15:47.172 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-04T08:15:47.173 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:15:47.247 INFO:teuthology.orchestra.run.smithi196:> true 2020-06-04T08:15:47.436 INFO:teuthology.orchestra.run.smithi196:> uname -m 2020-06-04T08:15:47.483 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2020-06-04T08:15:47.484 INFO:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2020-06-04T08:15:47.535 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="Ubuntu" 2020-06-04T08:15:47.536 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="18.04.4 LTS (Bionic Beaver)" 2020-06-04T08:15:47.536 INFO:teuthology.orchestra.run.smithi196.stdout:ID=ubuntu 2020-06-04T08:15:47.536 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE=debian 2020-06-04T08:15:47.536 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="Ubuntu 18.04.4 LTS" 2020-06-04T08:15:47.536 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="18.04" 2020-06-04T08:15:47.537 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://www.ubuntu.com/" 2020-06-04T08:15:47.537 INFO:teuthology.orchestra.run.smithi196.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-06-04T08:15:47.537 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-06-04T08:15:47.537 INFO:teuthology.orchestra.run.smithi196.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-06-04T08:15:47.537 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_CODENAME=bionic 2020-06-04T08:15:47.538 INFO:teuthology.orchestra.run.smithi196.stdout:UBUNTU_CODENAME=bionic 2020-06-04T08:15:47.546 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2020-06-04T08:15:47.566 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-06-04T08:15:47.612 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi060.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2sBIojSjMzey9aOmMk+qXyf+3XXUKYC8z2s2P34SimG2IxQrlBKbYii8YwiPqmDxwHXFwGe6emOAHi2Qbt5swScPnApnwLaqCisyP+CRaR48Oe1PgL/nV054P9J1dYeMbKnVdGwSABk6R3cM6e5Na3et3aDMH3L/ftTdxo4wToKWEbQCCqpgK4s2TEbKtGQ6CBuNtHtqBoV65ni853lz6jOLNfIc1RDcaNCeMeJsUrtggxjrQ8j/wxjjHqzw4PbszWwbv8sF6WBsFKQFcoA4CYliBVWzL5/BDK+FTMk0kAG4YX3FV4YYH+yPBFuAxLBqdkx5nfegBuP4xHSHpJ31b smithi196.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDejCVS35mBeiMVuY30POoD2qFLttOeNDZSkW0Byb9L8vUr1j0Thf72EQ2VqZ0FKgkgkADDw2bY7nNUZYxRfjTGLJNwr66E8oNgT82C44Muwli1+YU7JIkaFAVNPxrE28N1dOjdApLLT9i0W8pQOifHPKlvMWoAPMm6V8U5RDbTWdQ6gV0oq1SPlO+PNgCmV8UG7YQ02w/YXJcRgS0mNGRGqSJi1UMaVwQsZvNv4+OI6hOzjP5kxmpt7k2ThLv4CmmlzHfE6Pzb/Tv7rw9R4Coq/EFsTuzSbt2NShPUbgNX66BDX7kNZmYt8p372zyYfQZGUjDzyXXDC1eSuNiBJVhn 2020-06-04T08:15:47.612 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-06-04T08:15:47.690 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-06-04T08:15:47.699 INFO:teuthology.task.internal:Saving configuration 2020-06-04T08:15:47.735 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-06-04T08:15:47.817 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-06-04T08:15:47.835 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-06-04 08:10:28.547705', 'description': '/home/teuthworker/archive/yuvalif-2020-06-04_08:07:49-rgw:multisite-notifications_persistent_delivery2-distro-basic-smithi/5116651', 'locked_by': 'scheduled_yuvalif@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGLBpjuLqmtL69ybDG7xsIb/5dW6FAqcMwfHHpX96uln', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi060.front.sepia.ceph.com'} 2020-06-04T08:15:47.861 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-06-04 08:10:28.547726', 'description': '/home/teuthworker/archive/yuvalif-2020-06-04_08:07:49-rgw:multisite-notifications_persistent_delivery2-distro-basic-smithi/5116651', 'locked_by': 'scheduled_yuvalif@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFGeUEl3tFeA3o/anzSeJGg7MijAACsc5l59zES7j0y4', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi196.front.sepia.ceph.com'} 2020-06-04T08:15:47.861 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-06-04T08:15:47.874 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2020-06-04T08:15:47.875 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2020-06-04T08:15:47.875 INFO:teuthology.run_tasks:Running task console_log... 2020-06-04T08:15:47.994 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f26b6f85ae8>, signals=[15]) 2020-06-04T08:15:47.996 INFO:teuthology.run_tasks:Running task internal.connect... 2020-06-04T08:15:48.006 INFO:teuthology.task.internal:Opening connections... 2020-06-04T08:15:48.007 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi060.front.sepia.ceph.com 2020-06-04T08:15:48.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:15:48.090 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi196.front.sepia.ceph.com 2020-06-04T08:15:48.091 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:15:48.176 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-06-04T08:15:48.187 INFO:teuthology.orchestra.run.smithi060:> uname -m 2020-06-04T08:15:48.296 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2020-06-04T08:15:48.298 INFO:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2020-06-04T08:15:48.353 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="Ubuntu" 2020-06-04T08:15:48.353 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="18.04.4 LTS (Bionic Beaver)" 2020-06-04T08:15:48.353 INFO:teuthology.orchestra.run.smithi060.stdout:ID=ubuntu 2020-06-04T08:15:48.354 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE=debian 2020-06-04T08:15:48.354 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="Ubuntu 18.04.4 LTS" 2020-06-04T08:15:48.354 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="18.04" 2020-06-04T08:15:48.354 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://www.ubuntu.com/" 2020-06-04T08:15:48.354 INFO:teuthology.orchestra.run.smithi060.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-06-04T08:15:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-06-04T08:15:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-06-04T08:15:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_CODENAME=bionic 2020-06-04T08:15:48.355 INFO:teuthology.orchestra.run.smithi060.stdout:UBUNTU_CODENAME=bionic 2020-06-04T08:15:48.356 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2020-06-04T08:15:48.386 INFO:teuthology.orchestra.run.smithi196:> uname -m 2020-06-04T08:15:48.397 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2020-06-04T08:15:48.398 INFO:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2020-06-04T08:15:48.443 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="Ubuntu" 2020-06-04T08:15:48.444 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="18.04.4 LTS (Bionic Beaver)" 2020-06-04T08:15:48.444 INFO:teuthology.orchestra.run.smithi196.stdout:ID=ubuntu 2020-06-04T08:15:48.444 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE=debian 2020-06-04T08:15:48.445 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="Ubuntu 18.04.4 LTS" 2020-06-04T08:15:48.445 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="18.04" 2020-06-04T08:15:48.445 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://www.ubuntu.com/" 2020-06-04T08:15:48.445 INFO:teuthology.orchestra.run.smithi196.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-06-04T08:15:48.445 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-06-04T08:15:48.446 INFO:teuthology.orchestra.run.smithi196.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-06-04T08:15:48.446 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_CODENAME=bionic 2020-06-04T08:15:48.446 INFO:teuthology.orchestra.run.smithi196.stdout:UBUNTU_CODENAME=bionic 2020-06-04T08:15:48.446 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2020-06-04T08:15:48.474 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-06-04T08:15:48.529 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-06-04T08:15:48.553 INFO:teuthology.task.internal:Checking for old test directory... 2020-06-04T08:15:48.554 INFO:teuthology.orchestra.run.smithi060:> test '!' -e /home/ubuntu/cephtest 2020-06-04T08:15:48.558 INFO:teuthology.orchestra.run.smithi196:> test '!' -e /home/ubuntu/cephtest 2020-06-04T08:15:48.571 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-06-04T08:15:48.585 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-06-04T08:15:48.586 INFO:teuthology.orchestra.run.smithi060:> test -z $(ls -A /var/lib/ceph) 2020-06-04T08:15:48.609 INFO:teuthology.orchestra.run.smithi196:> test -z $(ls -A /var/lib/ceph) 2020-06-04T08:15:48.636 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-06-04T08:15:48.713 INFO:teuthology.run_tasks:Running task kernel... 2020-06-04T08:15:48.744 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2020-06-04T08:15:48.745 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-06-04T08:15:48.745 INFO:teuthology.orchestra.run.smithi060:> uname -r 2020-06-04T08:15:48.760 INFO:teuthology.orchestra.run.smithi060.stdout:4.15.0-99-generic 2020-06-04T08:15:48.760 INFO:teuthology.task.kernel:Running kernel on smithi060: 4.15.0-99-generic 2020-06-04T08:15:48.761 INFO:teuthology.orchestra.run.smithi060:> sudo apt-get clean 2020-06-04T08:15:48.870 INFO:teuthology.orchestra.run.smithi060:> sudo apt-get update 2020-06-04T08:15:49.002 INFO:teuthology.orchestra.run.smithi060.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-06-04T08:15:49.126 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-06-04T08:15:49.209 INFO:teuthology.orchestra.run.smithi060.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-06-04T08:15:49.261 INFO:teuthology.orchestra.run.smithi060.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [727 kB] 2020-06-04T08:15:49.353 INFO:teuthology.orchestra.run.smithi060.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [474 kB] 2020-06-04T08:15:49.360 INFO:teuthology.orchestra.run.smithi060.stdout:Get:6 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [230 kB] 2020-06-04T08:15:49.370 INFO:teuthology.orchestra.run.smithi060.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [45.7 kB] 2020-06-04T08:15:49.371 INFO:teuthology.orchestra.run.smithi060.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [11.4 kB] 2020-06-04T08:15:49.371 INFO:teuthology.orchestra.run.smithi060.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [623 kB] 2020-06-04T08:15:49.383 INFO:teuthology.orchestra.run.smithi060.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [669 kB] 2020-06-04T08:15:49.391 INFO:teuthology.orchestra.run.smithi060.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [222 kB] 2020-06-04T08:15:49.394 INFO:teuthology.orchestra.run.smithi060.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [7,596 B] 2020-06-04T08:15:49.394 INFO:teuthology.orchestra.run.smithi060.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [4,288 B] 2020-06-04T08:15:49.395 INFO:teuthology.orchestra.run.smithi060.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [2,824 B] 2020-06-04T08:15:49.485 INFO:teuthology.orchestra.run.smithi060.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-06-04T08:15:49.604 INFO:teuthology.orchestra.run.smithi060.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [685 kB] 2020-06-04T08:15:49.900 INFO:teuthology.orchestra.run.smithi060.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [951 kB] 2020-06-04T08:15:49.993 INFO:teuthology.orchestra.run.smithi060.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [324 kB] 2020-06-04T08:15:50.019 INFO:teuthology.orchestra.run.smithi060.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [55.3 kB] 2020-06-04T08:15:50.023 INFO:teuthology.orchestra.run.smithi060.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [13.8 kB] 2020-06-04T08:15:50.024 INFO:teuthology.orchestra.run.smithi060.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,018 kB] 2020-06-04T08:15:50.083 INFO:teuthology.orchestra.run.smithi060.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,077 kB] 2020-06-04T08:15:50.114 INFO:teuthology.orchestra.run.smithi060.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [335 kB] 2020-06-04T08:15:50.125 INFO:teuthology.orchestra.run.smithi060.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [8,720 B] 2020-06-04T08:15:50.125 INFO:teuthology.orchestra.run.smithi060.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [15.7 kB] 2020-06-04T08:15:50.126 INFO:teuthology.orchestra.run.smithi060.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,384 B] 2020-06-04T08:15:50.680 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 7,759 kB in 2s (4,829 kB/s) 2020-06-04T08:15:52.066 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2020-06-04T08:15:52.112 INFO:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-06-04T08:15:52.192 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2020-06-04T08:15:52.453 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2020-06-04T08:15:52.454 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2020-06-04T08:15:52.542 INFO:teuthology.orchestra.run.smithi060.stderr:E: Unable to locate package linux-image-current-generic 2020-06-04T08:15:52.544 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-06-04T08:15:52.545 INFO:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-06-04T08:15:52.633 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2020-06-04T08:15:52.864 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2020-06-04T08:15:52.865 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2020-06-04T08:15:53.044 INFO:teuthology.orchestra.run.smithi060.stdout:The following additional packages will be installed: 2020-06-04T08:15:53.045 INFO:teuthology.orchestra.run.smithi060.stdout: linux-generic linux-headers-4.15.0-101 linux-headers-4.15.0-101-generic 2020-06-04T08:15:53.045 INFO:teuthology.orchestra.run.smithi060.stdout: linux-headers-generic linux-image-4.15.0-101-generic 2020-06-04T08:15:53.045 INFO:teuthology.orchestra.run.smithi060.stdout: linux-modules-4.15.0-101-generic linux-modules-extra-4.15.0-101-generic 2020-06-04T08:15:53.046 INFO:teuthology.orchestra.run.smithi060.stdout:Suggested packages: 2020-06-04T08:15:53.046 INFO:teuthology.orchestra.run.smithi060.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2020-06-04T08:15:53.046 INFO:teuthology.orchestra.run.smithi060.stdout:Recommended packages: 2020-06-04T08:15:53.047 INFO:teuthology.orchestra.run.smithi060.stdout: thermald 2020-06-04T08:15:53.081 INFO:teuthology.orchestra.run.smithi060.stdout:The following NEW packages will be installed: 2020-06-04T08:15:53.081 INFO:teuthology.orchestra.run.smithi060.stdout: linux-headers-4.15.0-101 linux-headers-4.15.0-101-generic 2020-06-04T08:15:53.081 INFO:teuthology.orchestra.run.smithi060.stdout: linux-image-4.15.0-101-generic linux-modules-4.15.0-101-generic 2020-06-04T08:15:53.082 INFO:teuthology.orchestra.run.smithi060.stdout: linux-modules-extra-4.15.0-101-generic 2020-06-04T08:15:53.082 INFO:teuthology.orchestra.run.smithi060.stdout:The following packages will be upgraded: 2020-06-04T08:15:53.083 INFO:teuthology.orchestra.run.smithi060.stdout: linux-generic linux-headers-generic linux-image-generic 2020-06-04T08:15:53.265 INFO:teuthology.orchestra.run.smithi060.stdout:3 upgraded, 5 newly installed, 0 to remove and 55 not upgraded. 2020-06-04T08:15:53.267 INFO:teuthology.orchestra.run.smithi060.stdout:Need to get 65.8 MB of archives. 2020-06-04T08:15:53.267 INFO:teuthology.orchestra.run.smithi060.stdout:After this operation, 333 MB of additional disk space will be used. 2020-06-04T08:15:53.267 INFO:teuthology.orchestra.run.smithi060.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-101-generic amd64 4.15.0-101.102 [13.0 MB] 2020-06-04T08:15:54.285 INFO:teuthology.orchestra.run.smithi060.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-101-generic amd64 4.15.0-101.102 [8,005 kB] 2020-06-04T08:15:54.473 INFO:teuthology.orchestra.run.smithi060.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-101-generic amd64 4.15.0-101.102 [32.8 MB] 2020-06-04T08:15:55.451 INFO:teuthology.orchestra.run.smithi060.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.101.91 [1,864 B] 2020-06-04T08:15:55.451 INFO:teuthology.orchestra.run.smithi060.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.101.91 [2,464 B] 2020-06-04T08:15:55.452 INFO:teuthology.orchestra.run.smithi060.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-101 all 4.15.0-101.102 [10.9 MB] 2020-06-04T08:15:55.771 INFO:teuthology.orchestra.run.smithi060.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-101-generic amd64 4.15.0-101.102 [1,100 kB] 2020-06-04T08:15:55.822 INFO:teuthology.orchestra.run.smithi060.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.101.91 [2,424 B] 2020-06-04T08:15:56.285 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 65.8 MB in 3s (24.2 MB/s) 2020-06-04T08:15:56.433 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package linux-modules-4.15.0-101-generic. 2020-06-04T08:15:56.472 INFO:teuthology.orchestra.run.smithi060.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 101994 files and directories currently installed.) 2020-06-04T08:15:56.474 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../0-linux-modules-4.15.0-101-generic_4.15.0-101.102_amd64.deb ... 2020-06-04T08:15:56.523 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-modules-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-04T08:15:58.245 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package linux-image-4.15.0-101-generic. 2020-06-04T08:15:58.257 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../1-linux-image-4.15.0-101-generic_4.15.0-101.102_amd64.deb ... 2020-06-04T08:15:58.352 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-image-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-04T08:15:58.864 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-101-generic. 2020-06-04T08:15:58.876 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-101-generic_4.15.0-101.102_amd64.deb ... 2020-06-04T08:15:58.915 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-modules-extra-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-04T08:16:02.617 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../3-linux-generic_4.15.0.101.91_amd64.deb ... 2020-06-04T08:16:02.742 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-generic (4.15.0.101.91) over (4.15.0.99.89) ... 2020-06-04T08:16:03.056 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.101.91_amd64.deb ... 2020-06-04T08:16:03.182 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-image-generic (4.15.0.101.91) over (4.15.0.99.89) ... 2020-06-04T08:16:03.498 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package linux-headers-4.15.0-101. 2020-06-04T08:16:03.510 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../5-linux-headers-4.15.0-101_4.15.0-101.102_all.deb ... 2020-06-04T08:16:03.540 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-headers-4.15.0-101 (4.15.0-101.102) ... 2020-06-04T08:16:08.214 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package linux-headers-4.15.0-101-generic. 2020-06-04T08:16:08.230 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../6-linux-headers-4.15.0-101-generic_4.15.0-101.102_amd64.deb ... 2020-06-04T08:16:08.267 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-headers-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-04T08:16:10.750 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.101.91_amd64.deb ... 2020-06-04T08:16:10.860 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking linux-headers-generic (4.15.0.101.91) over (4.15.0.99.89) ... 2020-06-04T08:16:11.207 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-headers-4.15.0-101 (4.15.0-101.102) ... 2020-06-04T08:16:11.352 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-headers-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-04T08:16:11.478 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-modules-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-04T08:16:12.031 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-headers-generic (4.15.0.101.91) ... 2020-06-04T08:16:12.157 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-image-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-04T08:16:12.610 INFO:teuthology.orchestra.run.smithi060.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-101-generic 2020-06-04T08:16:12.610 INFO:teuthology.orchestra.run.smithi060.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-101-generic 2020-06-04T08:16:12.742 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-modules-extra-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-04T08:16:13.236 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-image-generic (4.15.0.101.91) ... 2020-06-04T08:16:13.362 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up linux-generic (4.15.0.101.91) ... 2020-06-04T08:16:13.480 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for linux-image-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-04T08:16:13.677 INFO:teuthology.orchestra.run.smithi060.stdout:/etc/kernel/postinst.d/initramfs-tools: 2020-06-04T08:16:13.677 INFO:teuthology.orchestra.run.smithi060.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-101-generic 2020-06-04T08:16:29.479 INFO:teuthology.orchestra.run.smithi060.stdout:/etc/kernel/postinst.d/zz-update-grub: 2020-06-04T08:16:29.480 INFO:teuthology.orchestra.run.smithi060.stdout:Sourcing file `/etc/default/grub' 2020-06-04T08:16:29.521 INFO:teuthology.orchestra.run.smithi060.stdout:Generating grub configuration file ... 2020-06-04T08:16:29.847 INFO:teuthology.orchestra.run.smithi060.stdout:Found linux image: /boot/vmlinuz-4.15.0-101-generic 2020-06-04T08:16:29.880 INFO:teuthology.orchestra.run.smithi060.stdout:Found initrd image: /boot/initrd.img-4.15.0-101-generic 2020-06-04T08:16:30.202 INFO:teuthology.orchestra.run.smithi060.stdout:Found linux image: /boot/vmlinuz-4.15.0-99-generic 2020-06-04T08:16:30.243 INFO:teuthology.orchestra.run.smithi060.stdout:Found initrd image: /boot/initrd.img-4.15.0-99-generic 2020-06-04T08:16:31.639 INFO:teuthology.orchestra.run.smithi060.stdout:done 2020-06-04T08:16:32.058 INFO:teuthology.orchestra.run.smithi060:> dpkg -s linux-image-generic 2020-06-04T08:16:32.083 INFO:teuthology.orchestra.run.smithi060.stdout:Package: linux-image-generic 2020-06-04T08:16:32.083 INFO:teuthology.orchestra.run.smithi060.stdout:Status: install ok installed 2020-06-04T08:16:32.083 INFO:teuthology.orchestra.run.smithi060.stdout:Priority: optional 2020-06-04T08:16:32.083 INFO:teuthology.orchestra.run.smithi060.stdout:Section: kernel 2020-06-04T08:16:32.084 INFO:teuthology.orchestra.run.smithi060.stdout:Installed-Size: 16 2020-06-04T08:16:32.084 INFO:teuthology.orchestra.run.smithi060.stdout:Maintainer: Ubuntu Kernel Team 2020-06-04T08:16:32.084 INFO:teuthology.orchestra.run.smithi060.stdout:Architecture: amd64 2020-06-04T08:16:32.084 INFO:teuthology.orchestra.run.smithi060.stdout:Source: linux-meta 2020-06-04T08:16:32.084 INFO:teuthology.orchestra.run.smithi060.stdout:Version: 4.15.0.101.91 2020-06-04T08:16:32.085 INFO:teuthology.orchestra.run.smithi060.stdout:Depends: linux-image-4.15.0-101-generic, linux-modules-extra-4.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2020-06-04T08:16:32.085 INFO:teuthology.orchestra.run.smithi060.stdout:Recommends: thermald 2020-06-04T08:16:32.085 INFO:teuthology.orchestra.run.smithi060.stdout:Description: Generic Linux kernel image 2020-06-04T08:16:32.085 INFO:teuthology.orchestra.run.smithi060.stdout: This package will always depend on the latest generic kernel image 2020-06-04T08:16:32.085 INFO:teuthology.orchestra.run.smithi060.stdout: available. 2020-06-04T08:16:32.086 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-99-generic Expected: 4.15.0-101-generic 2020-06-04T08:16:32.087 INFO:teuthology.orchestra.run.smithi196:> uname -r 2020-06-04T08:16:32.095 INFO:teuthology.orchestra.run.smithi196.stdout:4.15.0-99-generic 2020-06-04T08:16:32.095 INFO:teuthology.task.kernel:Running kernel on smithi196: 4.15.0-99-generic 2020-06-04T08:16:32.096 INFO:teuthology.orchestra.run.smithi196:> sudo apt-get clean 2020-06-04T08:16:32.207 INFO:teuthology.orchestra.run.smithi196:> sudo apt-get update 2020-06-04T08:16:32.334 INFO:teuthology.orchestra.run.smithi196.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-06-04T08:16:32.457 INFO:teuthology.orchestra.run.smithi196.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-06-04T08:16:32.538 INFO:teuthology.orchestra.run.smithi196.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-06-04T08:16:32.594 INFO:teuthology.orchestra.run.smithi196.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [727 kB] 2020-06-04T08:16:32.686 INFO:teuthology.orchestra.run.smithi196.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [474 kB] 2020-06-04T08:16:32.696 INFO:teuthology.orchestra.run.smithi196.stdout:Get:6 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [230 kB] 2020-06-04T08:16:32.700 INFO:teuthology.orchestra.run.smithi196.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [45.7 kB] 2020-06-04T08:16:32.712 INFO:teuthology.orchestra.run.smithi196.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [11.4 kB] 2020-06-04T08:16:32.712 INFO:teuthology.orchestra.run.smithi196.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [623 kB] 2020-06-04T08:16:32.719 INFO:teuthology.orchestra.run.smithi196.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [669 kB] 2020-06-04T08:16:32.732 INFO:teuthology.orchestra.run.smithi196.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [222 kB] 2020-06-04T08:16:32.737 INFO:teuthology.orchestra.run.smithi196.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [4,288 B] 2020-06-04T08:16:32.737 INFO:teuthology.orchestra.run.smithi196.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [7,596 B] 2020-06-04T08:16:32.738 INFO:teuthology.orchestra.run.smithi196.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [2,824 B] 2020-06-04T08:16:32.796 INFO:teuthology.orchestra.run.smithi196.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-06-04T08:16:32.909 INFO:teuthology.orchestra.run.smithi196.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [951 kB] 2020-06-04T08:16:33.243 INFO:teuthology.orchestra.run.smithi196.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [685 kB] 2020-06-04T08:16:33.283 INFO:teuthology.orchestra.run.smithi196.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [324 kB] 2020-06-04T08:16:33.302 INFO:teuthology.orchestra.run.smithi196.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [55.3 kB] 2020-06-04T08:16:33.329 INFO:teuthology.orchestra.run.smithi196.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [13.8 kB] 2020-06-04T08:16:33.330 INFO:teuthology.orchestra.run.smithi196.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,018 kB] 2020-06-04T08:16:33.382 INFO:teuthology.orchestra.run.smithi196.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,077 kB] 2020-06-04T08:16:33.417 INFO:teuthology.orchestra.run.smithi196.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [335 kB] 2020-06-04T08:16:33.426 INFO:teuthology.orchestra.run.smithi196.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [15.7 kB] 2020-06-04T08:16:33.427 INFO:teuthology.orchestra.run.smithi196.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [8,720 B] 2020-06-04T08:16:33.427 INFO:teuthology.orchestra.run.smithi196.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,384 B] 2020-06-04T08:16:33.994 INFO:teuthology.orchestra.run.smithi196.stdout:Fetched 7,759 kB in 2s (4,949 kB/s) 2020-06-04T08:16:35.359 INFO:teuthology.orchestra.run.smithi196.stdout:Reading package lists... 2020-06-04T08:16:35.405 INFO:teuthology.orchestra.run.smithi196:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-06-04T08:16:35.482 INFO:teuthology.orchestra.run.smithi196.stdout:Reading package lists... 2020-06-04T08:16:35.718 INFO:teuthology.orchestra.run.smithi196.stdout:Building dependency tree... 2020-06-04T08:16:35.719 INFO:teuthology.orchestra.run.smithi196.stdout:Reading state information... 2020-06-04T08:16:35.793 INFO:teuthology.orchestra.run.smithi196.stderr:E: Unable to locate package linux-image-current-generic 2020-06-04T08:16:35.794 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-06-04T08:16:35.795 INFO:teuthology.orchestra.run.smithi196:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-06-04T08:16:35.873 INFO:teuthology.orchestra.run.smithi196.stdout:Reading package lists... 2020-06-04T08:16:36.101 INFO:teuthology.orchestra.run.smithi196.stdout:Building dependency tree... 2020-06-04T08:16:36.101 INFO:teuthology.orchestra.run.smithi196.stdout:Reading state information... 2020-06-04T08:16:36.262 INFO:teuthology.orchestra.run.smithi196.stdout:The following additional packages will be installed: 2020-06-04T08:16:36.262 INFO:teuthology.orchestra.run.smithi196.stdout: linux-generic linux-headers-4.15.0-101 linux-headers-4.15.0-101-generic 2020-06-04T08:16:36.263 INFO:teuthology.orchestra.run.smithi196.stdout: linux-headers-generic linux-image-4.15.0-101-generic 2020-06-04T08:16:36.263 INFO:teuthology.orchestra.run.smithi196.stdout: linux-modules-4.15.0-101-generic linux-modules-extra-4.15.0-101-generic 2020-06-04T08:16:36.267 INFO:teuthology.orchestra.run.smithi196.stdout:Suggested packages: 2020-06-04T08:16:36.267 INFO:teuthology.orchestra.run.smithi196.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2020-06-04T08:16:36.267 INFO:teuthology.orchestra.run.smithi196.stdout:Recommended packages: 2020-06-04T08:16:36.268 INFO:teuthology.orchestra.run.smithi196.stdout: thermald 2020-06-04T08:16:36.293 INFO:teuthology.orchestra.run.smithi196.stdout:The following NEW packages will be installed: 2020-06-04T08:16:36.293 INFO:teuthology.orchestra.run.smithi196.stdout: linux-headers-4.15.0-101 linux-headers-4.15.0-101-generic 2020-06-04T08:16:36.293 INFO:teuthology.orchestra.run.smithi196.stdout: linux-image-4.15.0-101-generic linux-modules-4.15.0-101-generic 2020-06-04T08:16:36.296 INFO:teuthology.orchestra.run.smithi196.stdout: linux-modules-extra-4.15.0-101-generic 2020-06-04T08:16:36.296 INFO:teuthology.orchestra.run.smithi196.stdout:The following packages will be upgraded: 2020-06-04T08:16:36.297 INFO:teuthology.orchestra.run.smithi196.stdout: linux-generic linux-headers-generic linux-image-generic 2020-06-04T08:16:36.482 INFO:teuthology.orchestra.run.smithi196.stdout:3 upgraded, 5 newly installed, 0 to remove and 55 not upgraded. 2020-06-04T08:16:36.483 INFO:teuthology.orchestra.run.smithi196.stdout:Need to get 65.8 MB of archives. 2020-06-04T08:16:36.483 INFO:teuthology.orchestra.run.smithi196.stdout:After this operation, 333 MB of additional disk space will be used. 2020-06-04T08:16:36.483 INFO:teuthology.orchestra.run.smithi196.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-101-generic amd64 4.15.0-101.102 [13.0 MB] 2020-06-04T08:16:37.476 INFO:teuthology.orchestra.run.smithi196.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-101-generic amd64 4.15.0-101.102 [8,005 kB] 2020-06-04T08:16:37.672 INFO:teuthology.orchestra.run.smithi196.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-101-generic amd64 4.15.0-101.102 [32.8 MB] 2020-06-04T08:16:38.745 INFO:teuthology.orchestra.run.smithi196.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.101.91 [1,864 B] 2020-06-04T08:16:38.745 INFO:teuthology.orchestra.run.smithi196.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.101.91 [2,464 B] 2020-06-04T08:16:38.746 INFO:teuthology.orchestra.run.smithi196.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-101 all 4.15.0-101.102 [10.9 MB] 2020-06-04T08:16:39.046 INFO:teuthology.orchestra.run.smithi196.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-101-generic amd64 4.15.0-101.102 [1,100 kB] 2020-06-04T08:16:39.076 INFO:teuthology.orchestra.run.smithi196.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.101.91 [2,424 B] 2020-06-04T08:16:39.530 INFO:teuthology.orchestra.run.smithi196.stdout:Fetched 65.8 MB in 3s (23.9 MB/s) 2020-06-04T08:16:39.690 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package linux-modules-4.15.0-101-generic. 2020-06-04T08:16:39.718 INFO:teuthology.orchestra.run.smithi196.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 101994 files and directories currently installed.) 2020-06-04T08:16:39.723 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../0-linux-modules-4.15.0-101-generic_4.15.0-101.102_amd64.deb ... 2020-06-04T08:16:39.772 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking linux-modules-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-04T08:16:41.384 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package linux-image-4.15.0-101-generic. 2020-06-04T08:16:41.398 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../1-linux-image-4.15.0-101-generic_4.15.0-101.102_amd64.deb ... 2020-06-04T08:16:41.509 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking linux-image-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-04T08:16:41.994 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-101-generic. 2020-06-04T08:16:41.997 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-101-generic_4.15.0-101.102_amd64.deb ... 2020-06-04T08:16:42.049 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking linux-modules-extra-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-04T08:16:45.979 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../3-linux-generic_4.15.0.101.91_amd64.deb ... 2020-06-04T08:16:46.116 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking linux-generic (4.15.0.101.91) over (4.15.0.99.89) ... 2020-06-04T08:16:46.434 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.101.91_amd64.deb ... 2020-06-04T08:16:46.552 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking linux-image-generic (4.15.0.101.91) over (4.15.0.99.89) ... 2020-06-04T08:16:46.904 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package linux-headers-4.15.0-101. 2020-06-04T08:16:46.917 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../5-linux-headers-4.15.0-101_4.15.0-101.102_all.deb ... 2020-06-04T08:16:46.946 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking linux-headers-4.15.0-101 (4.15.0-101.102) ... 2020-06-04T08:16:51.669 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package linux-headers-4.15.0-101-generic. 2020-06-04T08:16:51.687 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../6-linux-headers-4.15.0-101-generic_4.15.0-101.102_amd64.deb ... 2020-06-04T08:16:51.729 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking linux-headers-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-04T08:16:54.043 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.101.91_amd64.deb ... 2020-06-04T08:16:54.137 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking linux-headers-generic (4.15.0.101.91) over (4.15.0.99.89) ... 2020-06-04T08:16:54.478 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up linux-headers-4.15.0-101 (4.15.0-101.102) ... 2020-06-04T08:16:54.618 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up linux-headers-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-04T08:16:54.734 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up linux-modules-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-04T08:16:55.285 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up linux-headers-generic (4.15.0.101.91) ... 2020-06-04T08:16:55.395 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up linux-image-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-04T08:16:55.896 INFO:teuthology.orchestra.run.smithi196.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-101-generic 2020-06-04T08:16:55.896 INFO:teuthology.orchestra.run.smithi196.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-101-generic 2020-06-04T08:16:56.030 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up linux-modules-extra-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-04T08:16:56.556 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up linux-image-generic (4.15.0.101.91) ... 2020-06-04T08:16:56.708 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up linux-generic (4.15.0.101.91) ... 2020-06-04T08:16:56.833 INFO:teuthology.orchestra.run.smithi196.stdout:Processing triggers for linux-image-4.15.0-101-generic (4.15.0-101.102) ... 2020-06-04T08:16:57.035 INFO:teuthology.orchestra.run.smithi196.stdout:/etc/kernel/postinst.d/initramfs-tools: 2020-06-04T08:16:57.036 INFO:teuthology.orchestra.run.smithi196.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-101-generic 2020-06-04T08:17:12.638 INFO:teuthology.orchestra.run.smithi196.stdout:/etc/kernel/postinst.d/zz-update-grub: 2020-06-04T08:17:12.639 INFO:teuthology.orchestra.run.smithi196.stdout:Sourcing file `/etc/default/grub' 2020-06-04T08:17:12.675 INFO:teuthology.orchestra.run.smithi196.stdout:Generating grub configuration file ... 2020-06-04T08:17:13.009 INFO:teuthology.orchestra.run.smithi196.stdout:Found linux image: /boot/vmlinuz-4.15.0-101-generic 2020-06-04T08:17:13.050 INFO:teuthology.orchestra.run.smithi196.stdout:Found initrd image: /boot/initrd.img-4.15.0-101-generic 2020-06-04T08:17:13.359 INFO:teuthology.orchestra.run.smithi196.stdout:Found linux image: /boot/vmlinuz-4.15.0-99-generic 2020-06-04T08:17:13.393 INFO:teuthology.orchestra.run.smithi196.stdout:Found initrd image: /boot/initrd.img-4.15.0-99-generic 2020-06-04T08:17:14.791 INFO:teuthology.orchestra.run.smithi196.stdout:done 2020-06-04T08:17:15.212 INFO:teuthology.orchestra.run.smithi196:> dpkg -s linux-image-generic 2020-06-04T08:17:15.275 INFO:teuthology.orchestra.run.smithi196.stdout:Package: linux-image-generic 2020-06-04T08:17:15.276 INFO:teuthology.orchestra.run.smithi196.stdout:Status: install ok installed 2020-06-04T08:17:15.276 INFO:teuthology.orchestra.run.smithi196.stdout:Priority: optional 2020-06-04T08:17:15.277 INFO:teuthology.orchestra.run.smithi196.stdout:Section: kernel 2020-06-04T08:17:15.277 INFO:teuthology.orchestra.run.smithi196.stdout:Installed-Size: 16 2020-06-04T08:17:15.277 INFO:teuthology.orchestra.run.smithi196.stdout:Maintainer: Ubuntu Kernel Team 2020-06-04T08:17:15.288 INFO:teuthology.orchestra.run.smithi196.stdout:Architecture: amd64 2020-06-04T08:17:15.289 INFO:teuthology.orchestra.run.smithi196.stdout:Source: linux-meta 2020-06-04T08:17:15.289 INFO:teuthology.orchestra.run.smithi196.stdout:Version: 4.15.0.101.91 2020-06-04T08:17:15.290 INFO:teuthology.orchestra.run.smithi196.stdout:Depends: linux-image-4.15.0-101-generic, linux-modules-extra-4.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2020-06-04T08:17:15.290 INFO:teuthology.orchestra.run.smithi196.stdout:Recommends: thermald 2020-06-04T08:17:15.291 INFO:teuthology.orchestra.run.smithi196.stdout:Description: Generic Linux kernel image 2020-06-04T08:17:15.291 INFO:teuthology.orchestra.run.smithi196.stdout: This package will always depend on the latest generic kernel image 2020-06-04T08:17:15.291 INFO:teuthology.orchestra.run.smithi196.stdout: available. 2020-06-04T08:17:15.292 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-99-generic Expected: 4.15.0-101-generic 2020-06-04T08:17:15.293 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2020-06-04T08:17:15.293 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-06-04T08:17:15.294 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-06-04T08:17:15.294 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2020-06-04T08:17:15.295 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi060.front.sepia.ceph.com, path=None, version=distro) 2020-06-04T08:17:15.295 INFO:teuthology.orchestra.run.smithi060:> sudo apt-get clean 2020-06-04T08:17:15.363 INFO:teuthology.orchestra.run.smithi060:> sudo apt-get update 2020-06-04T08:17:15.503 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-06-04T08:17:15.615 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-06-04T08:17:15.698 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-06-04T08:17:15.787 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-06-04T08:17:17.365 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2020-06-04T08:17:17.413 INFO:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-06-04T08:17:17.492 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2020-06-04T08:17:17.752 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2020-06-04T08:17:17.753 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2020-06-04T08:17:17.840 INFO:teuthology.orchestra.run.smithi060.stderr:E: Unable to locate package linux-image-current-generic 2020-06-04T08:17:17.842 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-06-04T08:17:17.843 INFO:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-06-04T08:17:17.968 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2020-06-04T08:17:18.203 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2020-06-04T08:17:18.204 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2020-06-04T08:17:18.407 INFO:teuthology.orchestra.run.smithi060.stdout:linux-image-generic is already the newest version (4.15.0.101.91). 2020-06-04T08:17:18.407 INFO:teuthology.orchestra.run.smithi060.stdout:0 upgraded, 0 newly installed, 0 to remove and 55 not upgraded. 2020-06-04T08:17:18.414 INFO:teuthology.orchestra.run.smithi060:> dpkg -s linux-image-generic 2020-06-04T08:17:18.445 INFO:teuthology.orchestra.run.smithi060.stdout:Package: linux-image-generic 2020-06-04T08:17:18.445 INFO:teuthology.orchestra.run.smithi060.stdout:Status: install ok installed 2020-06-04T08:17:18.445 INFO:teuthology.orchestra.run.smithi060.stdout:Priority: optional 2020-06-04T08:17:18.446 INFO:teuthology.orchestra.run.smithi060.stdout:Section: kernel 2020-06-04T08:17:18.446 INFO:teuthology.orchestra.run.smithi060.stdout:Installed-Size: 16 2020-06-04T08:17:18.446 INFO:teuthology.orchestra.run.smithi060.stdout:Maintainer: Ubuntu Kernel Team 2020-06-04T08:17:18.446 INFO:teuthology.orchestra.run.smithi060.stdout:Architecture: amd64 2020-06-04T08:17:18.446 INFO:teuthology.orchestra.run.smithi060.stdout:Source: linux-meta 2020-06-04T08:17:18.447 INFO:teuthology.orchestra.run.smithi060.stdout:Version: 4.15.0.101.91 2020-06-04T08:17:18.447 INFO:teuthology.orchestra.run.smithi060.stdout:Depends: linux-image-4.15.0-101-generic, linux-modules-extra-4.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2020-06-04T08:17:18.447 INFO:teuthology.orchestra.run.smithi060.stdout:Recommends: thermald 2020-06-04T08:17:18.447 INFO:teuthology.orchestra.run.smithi060.stdout:Description: Generic Linux kernel image 2020-06-04T08:17:18.447 INFO:teuthology.orchestra.run.smithi060.stdout: This package will always depend on the latest generic kernel image 2020-06-04T08:17:18.448 INFO:teuthology.orchestra.run.smithi060.stdout: available. 2020-06-04T08:17:18.448 INFO:teuthology.orchestra.run.smithi060:> mktemp 2020-06-04T08:17:18.494 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.oMdw3dHryS 2020-06-04T08:17:18.495 INFO:teuthology.orchestra.run.smithi060:> sudo cp /boot/grub/grub.cfg /tmp/tmp.oMdw3dHryS 2020-06-04T08:17:18.555 INFO:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.oMdw3dHryS 2020-06-04T08:17:18.673 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.oMdw3dHryS is 8KB 2020-06-04T08:17:18.690 INFO:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.oMdw3dHryS 2020-06-04T08:17:18.697 INFO:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2020-06-04T08:17:18.762 INFO:teuthology.orchestra.run.smithi060:> sudo sh -c 'cat > /etc/grub.d/01_ceph_kernel' && sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-06-04T08:17:18.867 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-101-generic 2020-06-04T08:17:18.867 INFO:teuthology.orchestra.run.smithi060:> sudo update-grub 2020-06-04T08:17:20.164 INFO:teuthology.orchestra.run.smithi060.stderr:Sourcing file `/etc/default/grub' 2020-06-04T08:17:20.177 INFO:teuthology.orchestra.run.smithi060.stderr:Generating grub configuration file ... 2020-06-04T08:17:20.415 INFO:teuthology.orchestra.run.smithi060.stderr:Found linux image: /boot/vmlinuz-4.15.0-101-generic 2020-06-04T08:17:20.430 INFO:teuthology.orchestra.run.smithi060.stderr:Found initrd image: /boot/initrd.img-4.15.0-101-generic 2020-06-04T08:17:20.862 INFO:teuthology.orchestra.run.smithi060.stderr:Found linux image: /boot/vmlinuz-4.15.0-99-generic 2020-06-04T08:17:20.875 INFO:teuthology.orchestra.run.smithi060.stderr:Found initrd image: /boot/initrd.img-4.15.0-99-generic 2020-06-04T08:17:21.999 INFO:teuthology.orchestra.run.smithi060.stderr:done 2020-06-04T08:17:21.999 INFO:teuthology.orchestra.run.smithi060:> sudo shutdown -r now 2020-06-04T08:17:22.013 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2020-06-04T08:17:22.013 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi196.front.sepia.ceph.com, path=None, version=distro) 2020-06-04T08:17:22.013 INFO:teuthology.orchestra.run.smithi196:> sudo apt-get clean 2020-06-04T08:17:22.092 INFO:teuthology.orchestra.run.smithi196:> sudo apt-get update 2020-06-04T08:17:22.362 INFO:teuthology.orchestra.run.smithi196.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-06-04T08:17:22.363 INFO:teuthology.orchestra.run.smithi196.stdout:Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-06-04T08:17:22.430 INFO:teuthology.orchestra.run.smithi196.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-06-04T08:17:22.511 INFO:teuthology.orchestra.run.smithi196.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-06-04T08:17:24.144 INFO:teuthology.orchestra.run.smithi196.stdout:Reading package lists... 2020-06-04T08:17:24.196 INFO:teuthology.orchestra.run.smithi196:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-06-04T08:17:24.297 INFO:teuthology.orchestra.run.smithi196.stdout:Reading package lists... 2020-06-04T08:17:24.542 INFO:teuthology.orchestra.run.smithi196.stdout:Building dependency tree... 2020-06-04T08:17:24.542 INFO:teuthology.orchestra.run.smithi196.stdout:Reading state information... 2020-06-04T08:17:24.622 INFO:teuthology.orchestra.run.smithi196.stderr:E: Unable to locate package linux-image-current-generic 2020-06-04T08:17:24.623 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-06-04T08:17:24.623 INFO:teuthology.orchestra.run.smithi196:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-06-04T08:17:24.718 INFO:teuthology.orchestra.run.smithi196.stdout:Reading package lists... 2020-06-04T08:17:24.996 INFO:teuthology.orchestra.run.smithi196.stdout:Building dependency tree... 2020-06-04T08:17:24.997 INFO:teuthology.orchestra.run.smithi196.stdout:Reading state information... 2020-06-04T08:17:25.184 INFO:teuthology.orchestra.run.smithi196.stdout:linux-image-generic is already the newest version (4.15.0.101.91). 2020-06-04T08:17:25.184 INFO:teuthology.orchestra.run.smithi196.stdout:0 upgraded, 0 newly installed, 0 to remove and 55 not upgraded. 2020-06-04T08:17:25.185 INFO:teuthology.orchestra.run.smithi196:> dpkg -s linux-image-generic 2020-06-04T08:17:25.233 INFO:teuthology.orchestra.run.smithi196.stdout:Package: linux-image-generic 2020-06-04T08:17:25.234 INFO:teuthology.orchestra.run.smithi196.stdout:Status: install ok installed 2020-06-04T08:17:25.234 INFO:teuthology.orchestra.run.smithi196.stdout:Priority: optional 2020-06-04T08:17:25.234 INFO:teuthology.orchestra.run.smithi196.stdout:Section: kernel 2020-06-04T08:17:25.235 INFO:teuthology.orchestra.run.smithi196.stdout:Installed-Size: 16 2020-06-04T08:17:25.235 INFO:teuthology.orchestra.run.smithi196.stdout:Maintainer: Ubuntu Kernel Team 2020-06-04T08:17:25.235 INFO:teuthology.orchestra.run.smithi196.stdout:Architecture: amd64 2020-06-04T08:17:25.235 INFO:teuthology.orchestra.run.smithi196.stdout:Source: linux-meta 2020-06-04T08:17:25.235 INFO:teuthology.orchestra.run.smithi196.stdout:Version: 4.15.0.101.91 2020-06-04T08:17:25.236 INFO:teuthology.orchestra.run.smithi196.stdout:Depends: linux-image-4.15.0-101-generic, linux-modules-extra-4.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2020-06-04T08:17:25.236 INFO:teuthology.orchestra.run.smithi196.stdout:Recommends: thermald 2020-06-04T08:17:25.236 INFO:teuthology.orchestra.run.smithi196.stdout:Description: Generic Linux kernel image 2020-06-04T08:17:25.236 INFO:teuthology.orchestra.run.smithi196.stdout: This package will always depend on the latest generic kernel image 2020-06-04T08:17:25.237 INFO:teuthology.orchestra.run.smithi196.stdout: available. 2020-06-04T08:17:25.239 INFO:teuthology.orchestra.run.smithi196:> mktemp 2020-06-04T08:17:25.304 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.QEVHqyJrE2 2020-06-04T08:17:25.305 INFO:teuthology.orchestra.run.smithi196:> sudo cp /boot/grub/grub.cfg /tmp/tmp.QEVHqyJrE2 2020-06-04T08:17:25.370 INFO:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.QEVHqyJrE2 2020-06-04T08:17:25.546 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.QEVHqyJrE2 is 8KB 2020-06-04T08:17:25.598 INFO:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.QEVHqyJrE2 2020-06-04T08:17:25.608 INFO:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2020-06-04T08:17:25.673 INFO:teuthology.orchestra.run.smithi196:> sudo sh -c 'cat > /etc/grub.d/01_ceph_kernel' && sudo chmod 755 /etc/grub.d/01_ceph_kernel 2020-06-04T08:17:25.773 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-101-generic 2020-06-04T08:17:25.778 INFO:teuthology.orchestra.run.smithi196:> sudo update-grub 2020-06-04T08:17:27.139 INFO:teuthology.orchestra.run.smithi196.stderr:Sourcing file `/etc/default/grub' 2020-06-04T08:17:27.170 INFO:teuthology.orchestra.run.smithi196.stderr:Generating grub configuration file ... 2020-06-04T08:17:27.449 INFO:teuthology.orchestra.run.smithi196.stderr:Found linux image: /boot/vmlinuz-4.15.0-101-generic 2020-06-04T08:17:27.463 INFO:teuthology.orchestra.run.smithi196.stderr:Found initrd image: /boot/initrd.img-4.15.0-101-generic 2020-06-04T08:17:27.758 INFO:teuthology.orchestra.run.smithi196.stderr:Found linux image: /boot/vmlinuz-4.15.0-99-generic 2020-06-04T08:17:27.770 INFO:teuthology.orchestra.run.smithi196.stderr:Found initrd image: /boot/initrd.img-4.15.0-99-generic 2020-06-04T08:17:28.777 INFO:teuthology.orchestra.run.smithi196.stderr:done 2020-06-04T08:17:28.779 INFO:teuthology.orchestra.run.smithi196:> sudo shutdown -r now 2020-06-04T08:17:58.818 INFO:teuthology.misc:Re-opening connections... 2020-06-04T08:17:58.820 INFO:teuthology.misc:trying to connect to ubuntu@smithi060.front.sepia.ceph.com 2020-06-04T08:17:58.821 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-04T08:17:58.822 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:18:58.826 DEBUG:teuthology.orchestra.remote:timed out 2020-06-04T08:18:58.832 INFO:teuthology.misc:trying to connect to ubuntu@smithi196.front.sepia.ceph.com 2020-06-04T08:18:58.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-04T08:18:58.834 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:19:01.886 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2020-06-04T08:19:01.887 DEBUG:teuthology.misc:waited 63.06663179397583 2020-06-04T08:19:02.887 INFO:teuthology.misc:trying to connect to ubuntu@smithi060.front.sepia.ceph.com 2020-06-04T08:19:02.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-04T08:19:02.889 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:19:03.025 INFO:teuthology.orchestra.run.smithi060:> true 2020-06-04T08:19:03.344 DEBUG:teuthology.misc:waited 64.52356791496277 2020-06-04T08:19:04.345 INFO:teuthology.misc:trying to connect to ubuntu@smithi196.front.sepia.ceph.com 2020-06-04T08:19:04.345 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-04T08:19:04.346 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:19:04.962 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2020-06-04T08:19:04.962 DEBUG:teuthology.misc:waited 66.14222478866577 2020-06-04T08:19:05.965 INFO:teuthology.misc:trying to connect to ubuntu@smithi196.front.sepia.ceph.com 2020-06-04T08:19:05.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-04T08:19:05.966 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:19:05.976 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2020-06-04T08:19:05.976 DEBUG:teuthology.misc:waited 67.15601229667664 2020-06-04T08:19:06.979 INFO:teuthology.misc:trying to connect to ubuntu@smithi196.front.sepia.ceph.com 2020-06-04T08:19:06.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-04T08:19:06.981 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:19:06.990 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2020-06-04T08:19:06.990 DEBUG:teuthology.misc:waited 68.17003607749939 2020-06-04T08:19:08.003 INFO:teuthology.misc:trying to connect to ubuntu@smithi196.front.sepia.ceph.com 2020-06-04T08:19:08.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-04T08:19:08.004 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:19:08.010 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2020-06-04T08:19:08.011 DEBUG:teuthology.misc:waited 69.19052863121033 2020-06-04T08:19:09.012 INFO:teuthology.misc:trying to connect to ubuntu@smithi196.front.sepia.ceph.com 2020-06-04T08:19:09.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-04T08:19:09.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:19:09.017 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2020-06-04T08:19:09.018 DEBUG:teuthology.misc:waited 70.19741654396057 2020-06-04T08:19:10.019 INFO:teuthology.misc:trying to connect to ubuntu@smithi196.front.sepia.ceph.com 2020-06-04T08:19:10.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-04T08:19:10.021 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:19:10.299 INFO:teuthology.orchestra.run.smithi196:> true 2020-06-04T08:19:11.729 DEBUG:teuthology.misc:waited 72.90905499458313 2020-06-04T08:19:12.838 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2020-06-04T08:19:12.839 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.15.0-101-generic"... 2020-06-04T08:19:12.839 INFO:teuthology.orchestra.run.smithi060:> uname -r 2020-06-04T08:19:12.847 INFO:teuthology.orchestra.run.smithi060.stdout:4.15.0-101-generic 2020-06-04T08:19:12.848 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-101-generic vs 4.15.0-101-generic 2020-06-04T08:19:12.848 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-06-04T08:19:12.848 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2020-06-04T08:19:12.849 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.15.0-101-generic"... 2020-06-04T08:19:12.849 INFO:teuthology.orchestra.run.smithi196:> uname -r 2020-06-04T08:19:12.857 INFO:teuthology.orchestra.run.smithi196.stdout:4.15.0-101-generic 2020-06-04T08:19:12.857 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-101-generic vs 4.15.0-101-generic 2020-06-04T08:19:12.858 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-06-04T08:19:13.860 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2020-06-04T08:19:13.860 INFO:teuthology.orchestra.run.smithi060:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-06-04T08:19:13.899 INFO:teuthology.orchestra.run.smithi060.stdout:ttyS1 2020-06-04T08:19:13.919 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2020-06-04T08:19:13.919 INFO:teuthology.orchestra.run.smithi196:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-06-04T08:19:14.149 INFO:teuthology.orchestra.run.smithi196.stdout:ttyS1 2020-06-04T08:19:14.164 INFO:teuthology.run_tasks:Running task internal.base... 2020-06-04T08:19:14.174 INFO:teuthology.task.internal:Creating test directory... 2020-06-04T08:19:14.175 INFO:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-06-04T08:19:14.177 INFO:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-06-04T08:19:14.185 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-06-04T08:19:14.262 INFO:teuthology.run_tasks:Running task internal.archive... 2020-06-04T08:19:14.377 INFO:teuthology.task.internal:Creating archive directory... 2020-06-04T08:19:14.378 INFO:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-06-04T08:19:14.380 INFO:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-06-04T08:19:14.392 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-06-04T08:19:14.514 INFO:teuthology.task.internal:Enabling coredump saving... 2020-06-04T08:19:14.515 INFO:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-06-04T08:19:14.518 INFO:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-06-04T08:19:14.538 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-06-04T08:19:14.542 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-06-04T08:19:14.572 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-06-04T08:19:14.581 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-06-04T08:19:14.583 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-06-04T08:19:14.609 INFO:teuthology.task.internal:Configuring sudo... 2020-06-04T08:19:14.609 INFO:teuthology.orchestra.run.smithi060:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-06-04T08:19:14.616 INFO:teuthology.orchestra.run.smithi196:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-06-04T08:19:14.649 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-06-04T08:19:14.669 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-06-04T08:19:14.670 INFO:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-06-04T08:19:14.673 INFO:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-06-04T08:19:14.705 INFO:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-06-04T08:19:14.732 INFO:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-06-04T08:19:14.780 INFO:teuthology.orchestra.run.smithi060:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-06-04T08:19:14.868 INFO:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-06-04T08:19:14.881 INFO:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-06-04T08:19:14.928 INFO:teuthology.orchestra.run.smithi196:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-06-04T08:19:15.016 INFO:teuthology.orchestra.run.smithi060:> sudo service rsyslog restart 2020-06-04T08:19:15.022 INFO:teuthology.orchestra.run.smithi196:> sudo service rsyslog restart 2020-06-04T08:19:15.107 INFO:teuthology.run_tasks:Running task internal.timer... 2020-06-04T08:19:15.119 INFO:teuthology.task.internal:Starting timer... 2020-06-04T08:19:15.119 INFO:teuthology.run_tasks:Running task kernel... 2020-06-04T08:19:15.183 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2020-06-04T08:19:15.183 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-06-04T08:19:15.184 INFO:teuthology.orchestra.run.smithi060:> uname -r 2020-06-04T08:19:15.192 INFO:teuthology.orchestra.run.smithi060.stdout:4.15.0-101-generic 2020-06-04T08:19:15.193 INFO:teuthology.task.kernel:Running kernel on smithi060: 4.15.0-101-generic 2020-06-04T08:19:15.193 INFO:teuthology.orchestra.run.smithi060:> sudo apt-get clean 2020-06-04T08:19:15.285 INFO:teuthology.orchestra.run.smithi060:> sudo apt-get update 2020-06-04T08:19:15.454 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-06-04T08:19:15.559 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-06-04T08:19:15.640 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-06-04T08:19:15.721 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-06-04T08:19:17.697 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2020-06-04T08:19:17.745 INFO:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-06-04T08:19:17.843 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2020-06-04T08:19:18.075 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2020-06-04T08:19:18.076 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2020-06-04T08:19:18.163 INFO:teuthology.orchestra.run.smithi060.stderr:E: Unable to locate package linux-image-current-generic 2020-06-04T08:19:18.164 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-06-04T08:19:18.165 INFO:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-06-04T08:19:18.246 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2020-06-04T08:19:18.482 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2020-06-04T08:19:18.482 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2020-06-04T08:19:18.688 INFO:teuthology.orchestra.run.smithi060.stdout:linux-image-generic is already the newest version (4.15.0.101.91). 2020-06-04T08:19:18.688 INFO:teuthology.orchestra.run.smithi060.stdout:0 upgraded, 0 newly installed, 0 to remove and 55 not upgraded. 2020-06-04T08:19:18.689 INFO:teuthology.orchestra.run.smithi060:> dpkg -s linux-image-generic 2020-06-04T08:19:18.715 INFO:teuthology.orchestra.run.smithi060.stdout:Package: linux-image-generic 2020-06-04T08:19:18.716 INFO:teuthology.orchestra.run.smithi060.stdout:Status: install ok installed 2020-06-04T08:19:18.716 INFO:teuthology.orchestra.run.smithi060.stdout:Priority: optional 2020-06-04T08:19:18.716 INFO:teuthology.orchestra.run.smithi060.stdout:Section: kernel 2020-06-04T08:19:18.716 INFO:teuthology.orchestra.run.smithi060.stdout:Installed-Size: 16 2020-06-04T08:19:18.716 INFO:teuthology.orchestra.run.smithi060.stdout:Maintainer: Ubuntu Kernel Team 2020-06-04T08:19:18.717 INFO:teuthology.orchestra.run.smithi060.stdout:Architecture: amd64 2020-06-04T08:19:18.717 INFO:teuthology.orchestra.run.smithi060.stdout:Source: linux-meta 2020-06-04T08:19:18.717 INFO:teuthology.orchestra.run.smithi060.stdout:Version: 4.15.0.101.91 2020-06-04T08:19:18.717 INFO:teuthology.orchestra.run.smithi060.stdout:Depends: linux-image-4.15.0-101-generic, linux-modules-extra-4.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2020-06-04T08:19:18.717 INFO:teuthology.orchestra.run.smithi060.stdout:Recommends: thermald 2020-06-04T08:19:18.718 INFO:teuthology.orchestra.run.smithi060.stdout:Description: Generic Linux kernel image 2020-06-04T08:19:18.719 INFO:teuthology.orchestra.run.smithi060.stdout: This package will always depend on the latest generic kernel image 2020-06-04T08:19:18.719 INFO:teuthology.orchestra.run.smithi060.stdout: available. 2020-06-04T08:19:18.719 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-06-04T08:19:18.720 INFO:teuthology.orchestra.run.smithi196:> uname -r 2020-06-04T08:19:18.731 INFO:teuthology.orchestra.run.smithi196.stdout:4.15.0-101-generic 2020-06-04T08:19:18.731 INFO:teuthology.task.kernel:Running kernel on smithi196: 4.15.0-101-generic 2020-06-04T08:19:18.732 INFO:teuthology.orchestra.run.smithi196:> sudo apt-get clean 2020-06-04T08:19:18.854 INFO:teuthology.orchestra.run.smithi196:> sudo apt-get update 2020-06-04T08:19:19.052 INFO:teuthology.orchestra.run.smithi196.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-06-04T08:19:19.141 INFO:teuthology.orchestra.run.smithi196.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-06-04T08:19:19.221 INFO:teuthology.orchestra.run.smithi196.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-06-04T08:19:19.303 INFO:teuthology.orchestra.run.smithi196.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-06-04T08:19:19.731 INFO:teuthology.orchestra.run.smithi196.stdout:Fetched 74.6 kB in 1s (117 kB/s) 2020-06-04T08:19:21.302 INFO:teuthology.orchestra.run.smithi196.stdout:Reading package lists... 2020-06-04T08:19:21.350 INFO:teuthology.orchestra.run.smithi196:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-06-04T08:19:21.438 INFO:teuthology.orchestra.run.smithi196.stdout:Reading package lists... 2020-06-04T08:19:21.667 INFO:teuthology.orchestra.run.smithi196.stdout:Building dependency tree... 2020-06-04T08:19:21.668 INFO:teuthology.orchestra.run.smithi196.stdout:Reading state information... 2020-06-04T08:19:21.743 INFO:teuthology.orchestra.run.smithi196.stderr:E: Unable to locate package linux-image-current-generic 2020-06-04T08:19:21.744 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-06-04T08:19:21.746 INFO:teuthology.orchestra.run.smithi196:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-06-04T08:19:21.828 INFO:teuthology.orchestra.run.smithi196.stdout:Reading package lists... 2020-06-04T08:19:22.073 INFO:teuthology.orchestra.run.smithi196.stdout:Building dependency tree... 2020-06-04T08:19:22.074 INFO:teuthology.orchestra.run.smithi196.stdout:Reading state information... 2020-06-04T08:19:22.262 INFO:teuthology.orchestra.run.smithi196.stdout:linux-image-generic is already the newest version (4.15.0.101.91). 2020-06-04T08:19:22.263 INFO:teuthology.orchestra.run.smithi196.stdout:0 upgraded, 0 newly installed, 0 to remove and 55 not upgraded. 2020-06-04T08:19:22.264 INFO:teuthology.orchestra.run.smithi196:> dpkg -s linux-image-generic 2020-06-04T08:19:22.279 INFO:teuthology.orchestra.run.smithi196.stdout:Package: linux-image-generic 2020-06-04T08:19:22.279 INFO:teuthology.orchestra.run.smithi196.stdout:Status: install ok installed 2020-06-04T08:19:22.279 INFO:teuthology.orchestra.run.smithi196.stdout:Priority: optional 2020-06-04T08:19:22.280 INFO:teuthology.orchestra.run.smithi196.stdout:Section: kernel 2020-06-04T08:19:22.280 INFO:teuthology.orchestra.run.smithi196.stdout:Installed-Size: 16 2020-06-04T08:19:22.280 INFO:teuthology.orchestra.run.smithi196.stdout:Maintainer: Ubuntu Kernel Team 2020-06-04T08:19:22.280 INFO:teuthology.orchestra.run.smithi196.stdout:Architecture: amd64 2020-06-04T08:19:22.280 INFO:teuthology.orchestra.run.smithi196.stdout:Source: linux-meta 2020-06-04T08:19:22.288 INFO:teuthology.orchestra.run.smithi196.stdout:Version: 4.15.0.101.91 2020-06-04T08:19:22.288 INFO:teuthology.orchestra.run.smithi196.stdout:Depends: linux-image-4.15.0-101-generic, linux-modules-extra-4.15.0-101-generic, linux-firmware, intel-microcode, amd64-microcode 2020-06-04T08:19:22.288 INFO:teuthology.orchestra.run.smithi196.stdout:Recommends: thermald 2020-06-04T08:19:22.289 INFO:teuthology.orchestra.run.smithi196.stdout:Description: Generic Linux kernel image 2020-06-04T08:19:22.289 INFO:teuthology.orchestra.run.smithi196.stdout: This package will always depend on the latest generic kernel image 2020-06-04T08:19:22.289 INFO:teuthology.orchestra.run.smithi196.stdout: available. 2020-06-04T08:19:22.289 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-06-04T08:19:22.290 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2020-06-04T08:19:22.290 INFO:teuthology.orchestra.run.smithi060:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-06-04T08:19:22.306 INFO:teuthology.orchestra.run.smithi060.stdout:ttyS1 2020-06-04T08:19:22.436 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2020-06-04T08:19:22.437 INFO:teuthology.orchestra.run.smithi196:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-06-04T08:19:22.452 INFO:teuthology.orchestra.run.smithi196.stdout:ttyS1 2020-06-04T08:19:22.589 INFO:teuthology.run_tasks:Running task pcp... 2020-06-04T08:19:22.610 INFO:teuthology.run_tasks:Running task selinux... 2020-06-04T08:19:22.622 DEBUG:teuthology.task.selinux:Excluding smithi060: OS 'ubuntu' does not support SELinux 2020-06-04T08:19:22.623 DEBUG:teuthology.task.selinux:Excluding smithi196: OS 'ubuntu' does not support SELinux 2020-06-04T08:19:22.623 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-06-04T08:19:22.623 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2020-06-04T08:19:22.624 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-06-04T08:19:22.624 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-06-04T08:19:22.645 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2020-06-04T08:19:22.646 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-06-04T08:19:22.690 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-06-04T08:19:22.702 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi060.front.sepia.ceph.com,smithi196.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-06-04T08:19:23.124 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-06-04T08:19:24.168 INFO:teuthology.task.ansible.out: [WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-06-04T08:19:26.588 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-06-04T08:19:26.622 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:26 +0000 (0:00:00.090) 0:00:00.090 ********* 2020-06-04T08:19:26.644 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:26 +0000 (0:00:00.021) 0:00:00.111 ********* 2020-06-04T08:19:28.516 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi060.front.sepia.ceph.com] 2020-06-04T08:19:28.613 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:28 +0000 (0:00:01.969) 0:00:02.080 ********* 2020-06-04T08:19:28.634 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] 2020-06-04T08:19:28.746 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:28 +0000 (0:00:00.132) 0:00:02.213 ********* 2020-06-04T08:19:29.028 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-06-04T08:19:29.030 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-06-04T08:19:29.069 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:29 +0000 (0:00:00.323) 0:00:02.537 ********* 2020-06-04T08:19:29.078 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-06-04T08:19:29.115 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:29 +0000 (0:00:00.045) 0:00:02.583 ********* 2020-06-04T08:19:29.646 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** changed: [smithi196.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-06-04T08:19:29.699 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:29 +0000 (0:00:00.584) 0:00:03.167 ********* 2020-06-04T08:19:29.710 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-06-04T08:19:29.747 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:29 +0000 (0:00:00.047) 0:00:03.214 ********* 2020-06-04T08:19:30.168 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-06-04T08:19:30.171 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.072655", "end": "2020-06-04 08:19:30.050856", "rc": 0, "start": "2020-06-04 08:19:29.978201", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-06-04T08:19:30.220 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:30 +0000 (0:00:00.473) 0:00:03.688 ********* 2020-06-04T08:19:30.234 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.096190", "end": "2020-06-04 08:19:30.606936", "rc": 0, "start": "2020-06-04 08:19:30.510746", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-06-04T08:19:30.268 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:30 +0000 (0:00:00.048) 0:00:03.736 ********* 2020-06-04T08:19:30.669 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-06-04T08:19:30.671 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-04T08:19:30.708 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:30 +0000 (0:00:00.439) 0:00:04.175 ********* 2020-06-04T08:19:30.721 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-04T08:19:30.755 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:30 +0000 (0:00:00.046) 0:00:04.222 ********* 2020-06-04T08:19:31.873 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1591258770.7758162-211160896994279/source", "state": "file", "uid": 0} 2020-06-04T08:19:31.921 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:31 +0000 (0:00:01.165) 0:00:05.388 ********* 2020-06-04T08:19:31.938 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1591258770.726143-84825127400709/source", "state": "file", "uid": 0} 2020-06-04T08:19:31.980 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:31 +0000 (0:00:00.058) 0:00:05.447 ********* 2020-06-04T08:19:32.445 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtC 2020-06-04T08:19:32.448 INFO:teuthology.task.ansible.out:toHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-06-04T08:19:32.486 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-06-04T08:19:32.488 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-06-04T08:19:32.628 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-06-04T08:19:32.634 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-06-04T08:19:32.924 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:32 +0000 (0:00:00.944) 0:00:06.391 ********* 2020-06-04T08:19:32.963 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:32 +0000 (0:00:00.039) 0:00:06.430 ********* 2020-06-04T08:19:33.676 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi060.front.sepia.ceph.com] 2020-06-04T08:19:33.736 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] 2020-06-04T08:19:33.786 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-06-04T08:19:33.788 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:33 +0000 (0:00:00.823) 0:00:07.254 ********* 2020-06-04T08:19:33.890 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:33 +0000 (0:00:00.098) 0:00:07.353 ********* 2020-06-04T08:19:34.306 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-06-04T08:19:34.308 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.199921", "end": "2020-06-04 08:19:34.258304", "rc": 0, "start": "2020-06-04 08:19:34.058383", "stderr": "", "stderr_lines": [], "stdout": "36acb243999edacffcb85d4e38b209d6e3ce3636", "stdout_lines": ["36acb243999edacffcb85d4e38b209d6e3ce3636"]} 2020-06-04T08:19:34.352 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:34 +0000 (0:00:00.461) 0:00:07.814 ********* 2020-06-04T08:19:34.442 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.226390", "end": "2020-06-04 08:19:34.401315", "rc": 0, "start": "2020-06-04 08:19:34.174925", "stderr": "", "stderr_lines": [], "stdout": "36acb243999edacffcb85d4e38b209d6e3ce3636", "stdout_lines": ["36acb243999edacffcb85d4e38b209d6e3ce3636"]} 2020-06-04T08:19:34.495 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:34 +0000 (0:00:00.145) 0:00:07.959 ********* 2020-06-04T08:19:34.518 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.008981", "end": "2020-06-04 08:19:34.917648", "failed_when_result": false, "rc": 0, "start": "2020-06-04 08:19:34.908667", "stderr": "", "stderr_lines": [], "stdout": "f148133fb0c07a635fda93cc17e7dcc2a5d008b7", "stdout_lines": ["f148133fb0c07a635fda93cc17e7dcc2a5d008b7"]} 2020-06-04T08:19:34.577 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:34 +0000 (0:00:00.083) 0:00:08.043 ********* 2020-06-04T08:19:34.615 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:19:34.664 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:34 +0000 (0:00:00.077) 0:00:08.121 ********* 2020-06-04T08:19:34.675 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi196.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.015650", "end": "2020-06-04 08:19:34.554214", "failed_when_result": false, "rc": 0, "start": "2020-06-04 08:19:34.538564", "stderr": "", "stderr_lines": [], "stdout": "f148133fb0c07a635fda93cc17e7dcc2a5d008b7", "stdout_lines": ["f148133fb0c07a635fda93cc17e7dcc2a5d008b7"]} 2020-06-04T08:19:34.713 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:34 +0000 (0:00:00.058) 0:00:08.180 ********* 2020-06-04T08:19:34.723 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-06-04T08:19:34.725 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:19:34.762 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:34 +0000 (0:00:00.049) 0:00:08.230 ********* 2020-06-04T08:19:34.773 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:19:34.814 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:34 +0000 (0:00:00.048) 0:00:08.278 ********* 2020-06-04T08:19:34.824 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:19:34.866 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:34 +0000 (0:00:00.055) 0:00:08.334 ********* 2020-06-04T08:19:34.877 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:19:34.913 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:34 +0000 (0:00:00.046) 0:00:08.381 ********* 2020-06-04T08:19:34.921 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-06-04T08:19:34.923 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:19:34.961 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:34 +0000 (0:00:00.048) 0:00:08.429 ********* 2020-06-04T08:19:34.971 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:19:35.020 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:35 +0000 (0:00:00.058) 0:00:08.488 ********* 2020-06-04T08:19:35.029 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:19:35.076 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:35 +0000 (0:00:00.055) 0:00:08.543 ********* 2020-06-04T08:19:35.088 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:19:35.136 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:35 +0000 (0:00:00.060) 0:00:08.604 ********* 2020-06-04T08:19:35.180 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-06-04T08:19:35.182 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:19:35.219 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:35 +0000 (0:00:00.080) 0:00:08.685 ********* 2020-06-04T08:19:35.226 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-06-04T08:19:35.228 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-04T08:19:35.265 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:35 +0000 (0:00:00.048) 0:00:08.733 ********* 2020-06-04T08:19:35.370 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-04T08:19:35.403 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:35 +0000 (0:00:00.137) 0:00:08.870 ********* 2020-06-04T08:19:35.490 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-06-04T08:19:35.614 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-06-04T08:19:35.652 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-06-04T08:19:35.752 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-06-04T08:19:35.810 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-06-04T08:19:35.902 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-06-04T08:19:35.968 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2020-06-04T08:19:36.051 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2020-06-04T08:19:36.110 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-06-04T08:19:36.206 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-06-04T08:19:36.286 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-06-04T08:19:36.350 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-06-04T08:19:36.469 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-06-04T08:19:36.501 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-06-04T08:19:36.633 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-06-04T08:19:36.675 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-06-04T08:19:36.811 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-06-04T08:19:36.838 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-06-04T08:19:36.984 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-06-04T08:19:36.990 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-06-04T08:19:37.150 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-06-04T08:19:37.163 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-06-04T08:19:37.272 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'aemerson', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-06-04T08:19:37.317 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'aemerson', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-06-04T08:19:37.435 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-06-04T08:19:37.490 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-06-04T08:19:37.583 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-06-04T08:19:37.636 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-06-04T08:19:37.729 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-06-04T08:19:37.794 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-06-04T08:19:37.898 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-06-04T08:19:37.968 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-06-04T08:19:38.032 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-06-04T08:19:38.117 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-06-04T08:19:38.171 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2020-06-04T08:19:38.278 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2020-06-04T08:19:38.331 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-06-04T08:19:38.437 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-06-04T08:19:38.492 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-06-04T08:19:38.596 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-06-04T08:19:38.629 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-06-04T08:19:38.753 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-06-04T08:19:38.786 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-06-04T08:19:38.899 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-06-04T08:19:38.948 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-06-04T08:19:39.070 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-06-04T08:19:39.131 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-06-04T08:19:39.207 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-06-04T08:19:39.271 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-06-04T08:19:39.362 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-06-04T08:19:39.429 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-06-04T08:19:39.521 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-06-04T08:19:39.586 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-06-04T08:19:39.671 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-06-04T08:19:39.692 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-06-04T08:19:39.836 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-06-04T08:19:39.859 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-06-04T08:19:39.983 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-06-04T08:19:40.139 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-06-04T08:19:40.151 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-06-04T08:19:40.303 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-06-04T08:19:40.305 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-06-04T08:19:40.471 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-06-04T08:19:40.484 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-06-04T08:19:40.631 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-06-04T08:19:40.671 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-06-04T08:19:40.779 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-06-04T08:19:40.818 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-06-04T08:19:40.925 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-06-04T08:19:40.958 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-06-04T08:19:41.079 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-06-04T08:19:41.116 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-06-04T08:19:41.234 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-06-04T08:19:41.262 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-06-04T08:19:41.387 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-06-04T08:19:41.410 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-06-04T08:19:41.558 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-06-04T08:19:41.568 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-06-04T08:19:41.726 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-06-04T08:19:41.728 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-06-04T08:19:41.877 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-06-04T08:19:41.878 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-06-04T08:19:42.016 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-06-04T08:19:42.146 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-06-04T08:19:42.168 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-06-04T08:19:42.282 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-06-04T08:19:42.346 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-06-04T08:19:42.420 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-06-04T08:19:42.511 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-06-04T08:19:42.565 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-06-04T08:19:42.687 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-06-04T08:19:42.710 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-06-04T08:19:42.848 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-06-04T08:19:42.861 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-06-04T08:19:43.022 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-06-04T08:19:43.025 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-06-04T08:19:43.196 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-06-04T08:19:43.198 INFO:teuthology.task.ansible.out: 2020-06-04T08:19:43.208 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-06-04T08:19:43.358 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-06-04T08:19:43.359 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-06-04T08:19:43.499 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-06-04T08:19:43.531 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-06-04T08:19:43.638 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-06-04T08:19:43.691 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-06-04T08:19:43.834 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-06-04T08:19:43.842 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-06-04T08:19:43.982 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-06-04T08:19:44.020 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-06-04T08:19:44.209 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-06-04T08:19:44.211 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-06-04T08:19:44.383 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-06-04T08:19:44.394 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-06-04T08:19:44.542 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-06-04T08:19:44.564 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-06-04T08:19:44.682 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-06-04T08:19:44.725 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-06-04T08:19:44.824 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-06-04T08:19:44.897 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-06-04T08:19:44.979 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-06-04T08:19:45.054 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-06-04T08:19:45.128 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-06-04T08:19:45.204 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-06-04T08:19:45.272 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-06-04T08:19:45.365 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-06-04T08:19:45.427 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-06-04T08:19:45.530 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-06-04T08:19:45.563 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-06-04T08:19:45.672 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-06-04T08:19:45.704 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-06-04T08:19:45.824 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-06-04T08:19:45.847 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-06-04T08:19:45.979 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-06-04T08:19:46.015 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-06-04T08:19:46.145 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-06-04T08:19:46.187 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-06-04T08:19:46.284 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-06-04T08:19:46.316 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-06-04T08:19:46.419 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-06-04T08:19:46.476 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-06-04T08:19:46.559 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-06-04T08:19:46.656 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-06-04T08:19:46.710 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-06-04T08:19:46.795 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-06-04T08:19:46.869 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-06-04T08:19:46.944 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-06-04T08:19:46.952 INFO:teuthology.task.ansible.out: 2020-06-04T08:19:47.029 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-06-04T08:19:47.088 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-06-04T08:19:47.214 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-06-04T08:19:47.236 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-06-04T08:19:47.237 INFO:teuthology.task.ansible.out: 2020-06-04T08:19:47.357 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-06-04T08:19:47.400 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-06-04T08:19:47.538 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-06-04T08:19:47.549 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-06-04T08:19:47.679 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-06-04T08:19:47.732 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-06-04T08:19:47.883 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-06-04T08:19:47.919 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-06-04T08:19:48.058 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-06-04T08:19:48.069 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-06-04T08:19:48.197 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-06-04T08:19:48.251 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-06-04T08:19:48.336 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-06-04T08:19:48.410 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-06-04T08:19:48.479 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-06-04T08:19:48.582 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-06-04T08:19:48.633 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-06-04T08:19:48.752 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-06-04T08:19:48.786 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-06-04T08:19:48.918 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-06-04T08:19:48.935 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-06-04T08:19:49.105 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-06-04T08:19:49.249 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-06-04T08:19:49.272 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-06-04T08:19:49.434 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-06-04T08:19:49.465 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-06-04T08:19:49.606 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-06-04T08:19:49.771 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-06-04T08:19:49.776 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-06-04T08:19:49.952 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-06-04T08:19:49.974 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-06-04T08:19:50.103 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-06-04T08:19:50.106 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-06-04T08:19:50.276 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-06-04T08:19:50.287 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-06-04T08:19:50.424 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-06-04T08:19:50.446 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-06-04T08:19:50.587 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-06-04T08:19:50.609 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-06-04T08:19:50.741 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2020-06-04T08:19:50.759 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-06-04T08:19:50.902 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-06-04T08:19:50.909 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-06-04T08:19:51.059 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-06-04T08:19:51.102 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-06-04T08:19:51.202 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-06-04T08:19:51.254 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-06-04T08:19:51.352 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-06-04T08:19:51.416 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2020-06-04T08:19:51.521 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-06-04T08:19:51.569 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-06-04T08:19:51.671 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-06-04T08:19:51.713 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-06-04T08:19:51.810 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-06-04T08:19:51.854 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-06-04T08:19:51.959 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2020-06-04T08:19:51.997 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-06-04T08:19:52.109 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2020-06-04T08:19:52.142 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-06-04T08:19:52.257 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2020-06-04T08:19:52.301 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-06-04T08:19:52.418 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2020-06-04T08:19:52.483 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-06-04T08:19:52.563 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2020-06-04T08:19:52.638 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2020-06-04T08:19:52.700 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2020-06-04T08:19:52.790 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2020-06-04T08:19:52.866 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2020-06-04T08:19:52.953 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2020-06-04T08:19:53.030 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2020-06-04T08:19:53.141 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2020-06-04T08:19:53.184 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2020-06-04T08:19:53.187 INFO:teuthology.task.ansible.out: 2020-06-04T08:19:53.285 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2020-06-04T08:19:53.318 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2020-06-04T08:19:53.462 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2020-06-04T08:19:53.606 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2020-06-04T08:19:53.754 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2020-06-04T08:19:53.903 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2020-06-04T08:19:54.047 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2020-06-04T08:19:54.126 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2020-06-04T08:19:54.660 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2020-06-04T08:19:55.133 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "name": "prateek", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2020-06-04T08:19:55.289 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "name": "prateek", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2020-06-04T08:19:55.713 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "name": "jelopez", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2020-06-04T08:19:55.773 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "name": "jelopez", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2020-06-04T08:19:55.821 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:55 +0000 (0:00:20.418) 0:00:29.288 ********* 2020-06-04T08:19:55.902 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-06-04T08:19:55.946 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:55 +0000 (0:00:00.118) 0:00:29.407 ********* 2020-06-04T08:19:55.986 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:55 +0000 (0:00:00.038) 0:00:29.446 ********* 2020-06-04T08:19:55.990 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-06-04T08:19:55.993 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 2020-06-04T08:19:55.998 INFO:teuthology.task.ansible.out: 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAx 2020-06-04T08:19:56.002 INFO:teuthology.task.ansible.out:j+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8ea 2020-06-04T08:19:56.006 INFO:teuthology.task.ansible.out:c8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f261 2020-06-04T08:19:56.010 INFO:teuthology.task.ansible.out:02d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomai 2020-06-04T08:19:56.011 INFO:teuthology.task.ansible.out:wang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redha 2020-06-04T08:19:56.013 INFO:teuthology.task.ansible.out:t.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4Uk 2020-06-04T08:19:56.014 INFO:teuthology.task.ansible.out:ibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMiv 2020-06-04T08:19:56.018 INFO:teuthology.task.ansible.out:DNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "beng 2020-06-04T08:19:56.020 INFO:teuthology.task.ansible.out:land@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name" 2020-06-04T08:19:56.023 INFO:teuthology.task.ansible.out:: "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643 2020-06-04T08:19:56.024 INFO:teuthology.task.ansible.out:eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb 2020-06-04T08:19:56.026 INFO:teuthology.task.ansible.out:801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}]}, "changed": false} TASK [users : Create all users without sudo access.] *************************** 2020-06-04T08:19:56.062 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:56 +0000 (0:00:00.082) 0:00:29.529 ********* 2020-06-04T08:19:56.138 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:56 +0000 (0:00:00.072) 0:00:29.601 ********* 2020-06-04T08:19:56.219 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-06-04T08:19:56.222 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 2020-06-04T08:19:56.224 INFO:teuthology.task.ansible.out: 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAx 2020-06-04T08:19:56.226 INFO:teuthology.task.ansible.out:j+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8ea 2020-06-04T08:19:56.233 INFO:teuthology.task.ansible.out:c8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f261 2020-06-04T08:19:56.235 INFO:teuthology.task.ansible.out:02d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomai 2020-06-04T08:19:56.238 INFO:teuthology.task.ansible.out:wang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redha 2020-06-04T08:19:56.240 INFO:teuthology.task.ansible.out:t.com Q9j56CPPXipXScmKi57PlQ fb6 2020-06-04T08:19:56.241 INFO:teuthology.task.ansible.out:16603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMiv 2020-06-04T08:19:56.243 INFO:teuthology.task.ansible.out:DNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "beng 2020-06-04T08:19:56.245 INFO:teuthology.task.ansible.out:land@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name" 2020-06-04T08:19:56.251 INFO:teuthology.task.ansible.out:: "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643 2020-06-04T08:19:56.254 INFO:teuthology.task.ansible.out:eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb 2020-06-04T08:19:56.256 INFO:teuthology.task.ansible.out:801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}]}, "changed": false} 2020-06-04T08:19:56.283 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:56 +0000 (0:00:00.149) 0:00:29.751 ********* 2020-06-04T08:19:56.989 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* ok: [smithi060.front.sepia.ceph.com -> localhost] => {"after": "36acb243999edacffcb85d4e38b209d6e3ce3636", "attempts": 1, "before": "36acb243999edacffcb85d4e38b209d6e3ce3636", "changed": false, "remote_url_changed": false} 2020-06-04T08:19:57.054 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:57 +0000 (0:00:00.767) 0:00:30.518 ********* 2020-06-04T08:19:57.064 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com -> localhost] => {"after": "36acb243999edacffcb85d4e38b209d6e3ce3636", "attempts": 1, "before": "36acb243999edacffcb85d4e38b209d6e3ce3636", "changed": false, "remote_url_changed": false} 2020-06-04T08:19:57.130 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:19:57 +0000 (0:00:00.076) 0:00:30.594 ********* 2020-06-04T08:19:57.309 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-06-04T08:19:57.312 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-06-04T08:19:57.482 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-06-04T08:19:57.647 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-06-04T08:19:57.659 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-06-04T08:19:57.791 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2020-06-04T08:19:57.793 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2020-06-04T08:19:57.823 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-06-04T08:19:57.953 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-06-04T08:19:58.019 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-06-04T08:19:58.106 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2020-06-04T08:19:58.108 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-06-04T08:19:58.182 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2020-06-04T08:19:58.183 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2020-06-04T08:19:58.260 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-06-04T08:19:58.317 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-06-04T08:19:58.407 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2020-06-04T08:19:58.461 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2020-06-04T08:19:58.465 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-06-04T08:19:58.564 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-06-04T08:19:58.578 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:19:58.610 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-06-04T08:19:58.738 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2020-06-04T08:19:58.739 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2020-06-04T08:19:58.748 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'aemerson', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:19:58.767 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:19:58.776 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:19:58.802 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2020-06-04T08:19:58.942 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-06-04T08:19:58.952 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-06-04T08:19:58.984 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:19:59.100 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2020-06-04T08:19:59.102 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2020-06-04T08:19:59.121 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2020-06-04T08:19:59.126 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2020-06-04T08:19:59.145 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'aemerson', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:19:59.157 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:19:59.177 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:19:59.281 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-06-04T08:19:59.318 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-06-04T08:19:59.425 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-06-04T08:19:59.470 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2020-06-04T08:19:59.472 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2020-06-04T08:19:59.571 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2020-06-04T08:19:59.624 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-06-04T08:19:59.742 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-06-04T08:19:59.799 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-06-04T08:19:59.890 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-06-04T08:19:59.946 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2020-06-04T08:20:00.086 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-06-04T08:20:00.187 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-06-04T08:20:00.197 INFO:teuthology.task.ansible.out: 2020-06-04T08:20:00.280 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2020-06-04T08:20:00.282 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2020-06-04T08:20:00.342 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-06-04T08:20:00.490 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2020-06-04T08:20:00.495 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2020-06-04T08:20:00.505 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-06-04T08:20:00.639 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-06-04T08:20:00.671 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2020-06-04T08:20:00.673 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2020-06-04T08:20:00.799 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-06-04T08:20:00.898 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2020-06-04T08:20:00.902 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2020-06-04T08:20:01.004 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2020-06-04T08:20:01.006 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2020-06-04T08:20:01.061 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-06-04T08:20:01.145 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-06-04T08:20:01.209 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-06-04T08:20:01.322 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-06-04T08:20:01.371 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2020-06-04T08:20:01.374 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2020-06-04T08:20:01.470 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-06-04T08:20:01.524 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-06-04T08:20:01.650 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-06-04T08:20:01.677 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-06-04T08:20:01.821 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-06-04T08:20:01.832 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-06-04T08:20:01.979 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-06-04T08:20:02.035 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2020-06-04T08:20:02.037 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2020-06-04T08:20:02.131 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-06-04T08:20:02.210 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-06-04T08:20:02.290 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2020-06-04T08:20:02.295 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2020-06-04T08:20:02.380 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-06-04T08:20:02.447 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-06-04T08:20:02.533 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-06-04T08:20:02.607 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-06-04T08:20:02.702 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-06-04T08:20:02.761 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-06-04T08:20:02.867 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2020-06-04T08:20:02.898 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-06-04T08:20:03.046 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-06-04T08:20:03.050 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2020-06-04T08:20:03.193 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-06-04T08:20:03.238 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-06-04T08:20:03.355 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-06-04T08:20:03.412 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-06-04T08:20:03.491 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-06-04T08:20:03.602 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-06-04T08:20:03.626 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-06-04T08:20:03.754 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2020-06-04T08:20:03.756 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-06-04T08:20:03.799 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2020-06-04T08:20:03.801 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-06-04T08:20:03.920 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2020-06-04T08:20:03.922 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-06-04T08:20:03.944 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2020-06-04T08:20:03.946 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-06-04T08:20:04.070 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-06-04T08:20:04.102 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-06-04T08:20:04.266 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-06-04T08:20:04.292 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-06-04T08:20:04.430 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-06-04T08:20:04.458 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-06-04T08:20:04.602 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2020-06-04T08:20:04.605 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-06-04T08:20:04.629 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2020-06-04T08:20:04.632 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-06-04T08:20:04.769 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2020-06-04T08:20:04.772 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2020-06-04T08:20:04.774 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-06-04T08:20:04.854 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2020-06-04T08:20:04.856 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2020-06-04T08:20:04.862 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-06-04T08:20:04.954 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-06-04T08:20:05.037 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-06-04T08:20:05.086 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-06-04T08:20:05.181 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-06-04T08:20:05.238 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-06-04T08:20:05.346 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-06-04T08:20:05.374 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-06-04T08:20:05.513 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-06-04T08:20:05.534 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-06-04T08:20:05.682 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2020-06-04T08:20:05.688 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-06-04T08:20:05.701 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-06-04T08:20:05.829 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-06-04T08:20:05.930 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2020-06-04T08:20:05.934 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-06-04T08:20:05.991 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-06-04T08:20:06.104 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-06-04T08:20:06.158 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2020-06-04T08:20:06.160 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2020-06-04T08:20:06.279 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-06-04T08:20:06.312 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-06-04T08:20:06.441 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2020-06-04T08:20:06.443 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2020-06-04T08:20:06.491 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-06-04T08:20:06.612 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-06-04T08:20:06.628 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2020-06-04T08:20:06.630 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2020-06-04T08:20:06.757 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-06-04T08:20:06.774 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-06-04T08:20:06.919 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2020-06-04T08:20:06.921 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzn 2020-06-04T08:20:06.923 INFO:teuthology.task.ansible.out:er/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-06-04T08:20:07.082 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-06-04T08:20:07.094 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-06-04T08:20:07.247 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-06-04T08:20:07.270 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-06-04T08:20:07.396 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-06-04T08:20:07.443 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-06-04T08:20:07.551 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-06-04T08:20:07.623 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-06-04T08:20:07.721 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-06-04T08:20:07.767 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-06-04T08:20:07.886 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-06-04T08:20:07.967 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-06-04T08:20:08.067 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-06-04T08:20:08.123 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-06-04T08:20:08.204 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-06-04T08:20:08.266 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-06-04T08:20:08.356 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-06-04T08:20:08.444 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-06-04T08:20:08.507 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-06-04T08:20:08.606 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-06-04T08:20:08.657 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-06-04T08:20:08.770 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-06-04T08:20:08.822 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-06-04T08:20:08.937 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-06-04T08:20:09.008 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-06-04T08:20:09.126 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-06-04T08:20:09.194 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-06-04T08:20:09.301 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-06-04T08:20:09.334 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-06-04T08:20:09.447 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-06-04T08:20:09.518 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-06-04T08:20:09.614 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-06-04T08:20:09.674 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-06-04T08:20:09.778 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-06-04T08:20:09.822 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-06-04T08:20:09.921 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-06-04T08:20:09.982 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-06-04T08:20:10.083 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-06-04T08:20:10.139 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-06-04T08:20:10.268 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-06-04T08:20:10.290 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-06-04T08:20:10.385 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-06-04T08:20:10.438 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-06-04T08:20:10.560 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-06-04T08:20:10.602 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2020-06-04T08:20:10.702 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-06-04T08:20:10.748 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2020-06-04T08:20:10.762 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-06-04T08:20:10.848 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-06-04T08:20:10.906 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2020-06-04T08:20:10.908 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-06-04T08:20:11.014 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2020-06-04T08:20:11.043 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-06-04T08:20:11.188 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2020-06-04T08:20:11.190 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-06-04T08:20:11.226 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-06-04T08:20:11.364 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2020-06-04T08:20:11.366 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-06-04T08:20:11.375 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-06-04T08:20:11.505 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-06-04T08:20:11.537 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-06-04T08:20:11.669 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-06-04T08:20:11.696 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-06-04T08:20:11.841 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-06-04T08:20:11.850 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-06-04T08:20:12.005 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-06-04T08:20:12.058 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-06-04T08:20:12.166 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-06-04T08:20:12.251 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-06-04T08:20:12.356 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-06-04T08:20:12.417 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-06-04T08:20:12.525 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-06-04T08:20:12.588 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-06-04T08:20:12.720 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-06-04T08:20:12.798 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-06-04T08:20:12.883 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-06-04T08:20:12.967 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-06-04T08:20:13.038 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-06-04T08:20:13.126 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-06-04T08:20:13.250 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-06-04T08:20:13.298 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-06-04T08:20:13.383 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2020-06-04T08:20:13.397 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-06-04T08:20:13.467 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-06-04T08:20:13.555 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-06-04T08:20:13.632 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-06-04T08:20:13.721 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-06-04T08:20:13.803 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-06-04T08:20:13.865 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2020-06-04T08:20:13.949 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2020-06-04T08:20:13.951 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-06-04T08:20:14.027 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-06-04T08:20:14.119 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-06-04T08:20:14.179 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzVa77kg0qGGE3IpMZgSYJ+UwQiDEOg3Lji470V/sQ/R7g6/v7FZpaYqGn1wG3XeUp2gGACxNKLQfGV9EGU66NvfQYqn9A8dXRpTpZuGWN8EcKK98YrSAq0s0WzeiEzVT86RHzVHIqL6/Zn4B1p9sIeY9Hb+bfKOdtYkRBDoFELtiMRzHPFaiIJFnXl+EhH8wceYqXVNNvCuOUBMnOiDYLDtujvYdPt+ff7rvXuOzbwcAKoINFM1C4FtoUEYriKlsxJ2rr7AblfDm/WIrCZcqRBU4JcudtvZOtHhvN4uL6WgzuzuhIQFyePAeIHFUn6AiwKC/MApq7z1Qj+N3pN/Nz xiubli@fedora1\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkS3qZAsTQMObkUS9QBEpYlL7A3/4ypeRiVQil7UXrZkXVhYd8RAlvSBomOHtslz6OMuvm1GtP53apQQbLb2ziJXlogua/2ju17iyBP8KfyGUzx45S3XF6GaSi3C4gyviQ0hLhgg0HSaO92qw7lbjf731hXYBWXDwkXz/6wbYecykKHTM2xeIhTqCSDIx+fdrHmYW8tXp5ZIGxmCXgLhyNLaMXlNrsGHvLixrnUb3abJebdi+diQxWn4fx2Q+WwoAnY1uwawyeFJzmLEs3qjFWmuYSKcDJ94gHHedXyDhZN1mZOs0a0E319jtXG88/SQrX3+/OypS92Hr0FF6PQwhT root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-06-04T08:20:14.286 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-06-04T08:20:14.328 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-06-04T08:20:14.438 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2020-06-04T08:20:14.482 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2020-06-04T08:20:14.623 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-06-04T08:20:14.655 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2020-06-04T08:20:14.784 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzVa77kg0qGGE3IpMZgSYJ+UwQiDEOg3Lji470V/sQ/R7g6/v7FZpaYqGn1wG3XeUp2gGACxNKLQfGV9EGU66NvfQYqn9A8dXRpTpZuGWN8EcKK98YrSAq0s0WzeiEzVT86RHzVHIqL6/Zn4B1p9sIeY9Hb+bfKOdtYkRBDoFELtiMRzHPFaiIJFnXl+EhH8wceYqXVNNvCuOUBMnOiDYLDtujvYdPt+ff7rvXuOzbwcAKoINFM1C4FtoUEYriKlsxJ2rr7AblfDm/WIrCZcqRBU4JcudtvZOtHhvN4uL6WgzuzuhIQFyePAeIHFUn6AiwKC/MApq7z1Qj+N3pN/Nz xiubli@fedora1\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkS3qZAsTQMObkUS9QBEpYlL7A3/4ypeRiVQil7UXrZkXVhYd8RAlvSBomOHtslz6OMuvm1GtP53apQQbLb2ziJXlogua/2ju17iyBP8KfyGUzx45S3XF6GaSi3C4gyviQ0hLhgg0HSaO92qw7lbjf731hXYBWXDwkXz/6wbYecykKHTM2xeIhTqCSDIx+fdrHmYW8tXp5ZIGxmCXgLhyNLaMXlNrsGHvLixrnUb3abJebdi+diQxWn4fx2Q+WwoAnY1uwawyeFJzmLEs3qjFWmuYSKcDJ94gHHedXyDhZN1mZOs0a0E319jtXG88/SQrX3+/OypS92Hr0FF6PQwhT root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-06-04T08:20:14.810 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2020-06-04T08:20:14.944 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-06-04T08:20:14.965 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2020-06-04T08:20:15.104 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2020-06-04T08:20:15.109 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2020-06-04T08:20:15.243 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2020-06-04T08:20:15.256 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2020-06-04T08:20:15.388 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2020-06-04T08:20:15.411 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2020-06-04T08:20:15.530 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2020-06-04T08:20:15.562 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2020-06-04T08:20:15.669 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2020-06-04T08:20:15.692 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2020-06-04T08:20:15.810 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2020-06-04T08:20:15.876 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2020-06-04T08:20:15.923 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:15 +0000 (0:00:18.796) 0:00:49.391 ********* 2020-06-04T08:20:15.984 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:15.999 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.023 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.024 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.036 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-06-04T08:20:16.038 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2020-06-04T08:20:16.050 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.064 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.075 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.123 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.174 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi060.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2020-06-04T08:20:16.325 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2020-06-04T08:20:16.335 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.344 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-06-04T08:20:16.347 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2020-06-04T08:20:16.501 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-06-04T08:20:16.503 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'aemerson', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2020-06-04T08:20:16.516 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi060.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2020-06-04T08:20:16.642 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:16 +0000 (0:00:00.717) 0:00:50.109 ********* 2020-06-04T08:20:16.676 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-06-04T08:20:16.709 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.721 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.732 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.743 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.754 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.756 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.779 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.795 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.811 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2020-06-04T08:20:16.834 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.837 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.846 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.857 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.870 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.872 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.882 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.894 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.896 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.912 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.918 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.927 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.929 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2020-06-04T08:20:16.941 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.956 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.958 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.971 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.972 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.984 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.996 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:16.998 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.010 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.022 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.024 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.033 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.044 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.046 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.056 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.067 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.088 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.090 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.100 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.104 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.111 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'aemerson', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2020-06-04T08:20:17.113 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.124 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.125 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.144 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.146 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.156 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.172 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.183 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.185 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.205 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.207 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.217 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.229 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.242 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.255 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.272 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.287 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.290 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.343 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-06-04T08:20:17.352 INFO:teuthology.task.ansible.out: skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.354 INFO:teuthology.task.ansible.out: skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.355 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.355 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.356 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": 2020-06-04T08:20:17.358 INFO:teuthology.task.ansible.out:"bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.392 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.415 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} ok: [smithi060.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBp 2020-06-04T08:20:17.418 INFO:teuthology.task.ansible.out:gc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.426 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.437 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fed 2020-06-04T08:20:17.441 INFO:teuthology.task.ansible.out:oragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.449 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.450 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.462 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.464 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.478 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.497 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.499 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.508 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.532 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.533 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.552 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.555 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item= 2020-06-04T08:20:17.557 INFO:teuthology.task.ansible.out:{'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.568 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.580 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.582 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.591 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.593 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.602 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.604 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.614 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.616 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.626 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.628 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.640 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.645 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.658 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.660 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.673 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.676 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.678 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.680 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.682 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.695 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.697 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.711 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.714 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.723 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.724 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.726 INFO:teuthology.task.ansible.out: skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.728 INFO:teuthology.task.ansible.out: 2020-06-04T08:20:17.757 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.760 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.801 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:17 +0000 (0:00:01.159) 0:00:51.268 ********* 2020-06-04T08:20:17.811 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.813 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.838 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.897 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:17 +0000 (0:00:00.095) 0:00:51.364 ********* 2020-06-04T08:20:17.907 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.909 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.910 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.912 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.913 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.921 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.922 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.949 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.951 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.965 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.966 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.967 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.968 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.969 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@red 2020-06-04T08:20:17.971 INFO:teuthology.task.ansible.out:hat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:17.993 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.002 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.025 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.026 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.039 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.050 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.062 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.083 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.096 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.099 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.108 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.111 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi196.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2020-06-04T08:20:18.121 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-06-04T08:20:18.122 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.133 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.156 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.178 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.192 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.208 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.215 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.226 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.243 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.255 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-04T08:20:18.256 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-06-04T08:20:18.266 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-06-04T08:20:18.268 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.279 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.292 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.313 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.315 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.329 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.342 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.354 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.385 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.399 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.414 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-04T08:20:18.415 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.462 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.470 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d\nxiubli@lxbceph0 s0gvBwazopcgx4LPu0pQiA 1b2012fe8c6ea0a54dfd0e7a6fb1a19982b934cf6d84a64d21aa16dbe8510db3\n"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.478 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.489 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.494 INFO:teuthology.task.ansible.out: 2020-06-04T08:20:18.503 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.527 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.528 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.539 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.540 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.541 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi196.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-06-04T08:20:18.542 INFO:teuthology.task.ansible.out: 2020-06-04T08:20:18.544 INFO:teuthology.task.ansible.out:TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.559 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.580 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.582 INFO:teuthology.task.ansible.out: 2020-06-04T08:20:18.652 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:18 +0000 (0:00:00.755) 0:00:52.119 ********* 2020-06-04T08:20:18.673 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-04T08:20:18.675 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-06-04T08:20:18.700 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:18.739 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:18 +0000 (0:00:00.087) 0:00:52.207 ********* 2020-06-04T08:20:18.835 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-04T08:20:18.837 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-06-04T08:20:18.914 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2020-06-04T08:20:18.966 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2020-06-04T08:20:19.055 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-06-04T08:20:19.118 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2020-06-04T08:20:19.210 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-06-04T08:20:19.246 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2020-06-04T08:20:19.352 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-06-04T08:20:19.374 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-06-04T08:20:19.503 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-06-04T08:20:19.525 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-06-04T08:20:19.643 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-06-04T08:20:19.666 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-06-04T08:20:19.791 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2020-06-04T08:20:19.825 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-06-04T08:20:19.944 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2020-06-04T08:20:19.950 INFO:teuthology.task.ansible.out: 2020-06-04T08:20:19.969 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-06-04T08:20:20.087 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2020-06-04T08:20:20.109 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-06-04T08:20:20.224 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-06-04T08:20:20.272 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-06-04T08:20:20.364 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-06-04T08:20:20.417 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-06-04T08:20:20.522 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-06-04T08:20:20.557 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-06-04T08:20:20.657 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-06-04T08:20:20.700 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-06-04T08:20:20.797 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-06-04T08:20:20.839 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-06-04T08:20:20.936 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-06-04T08:20:20.988 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-06-04T08:20:21.082 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-06-04T08:20:21.115 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-06-04T08:20:21.235 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-06-04T08:20:21.267 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-06-04T08:20:21.390 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-06-04T08:20:21.392 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-06-04T08:20:21.529 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-06-04T08:20:21.553 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-06-04T08:20:21.678 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-06-04T08:20:21.680 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-06-04T08:20:21.812 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-06-04T08:20:21.834 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-06-04T08:20:21.950 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-06-04T08:20:21.982 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-06-04T08:20:22.122 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-06-04T08:20:22.127 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-06-04T08:20:22.263 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-06-04T08:20:22.309 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-06-04T08:20:22.410 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-06-04T08:20:22.448 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-06-04T08:20:22.575 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-06-04T08:20:22.597 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-06-04T08:20:22.730 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-06-04T08:20:22.753 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-06-04T08:20:22.894 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-06-04T08:20:22.944 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-06-04T08:20:23.045 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-06-04T08:20:23.087 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2020-06-04T08:20:23.171 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-06-04T08:20:23.230 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-06-04T08:20:23.323 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-06-04T08:20:23.365 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-06-04T08:20:23.481 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-06-04T08:20:23.526 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-06-04T08:20:23.628 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-06-04T08:20:23.681 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-06-04T08:20:23.786 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-06-04T08:20:23.817 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-06-04T08:20:23.940 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2020-06-04T08:20:23.954 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-06-04T08:20:24.076 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-06-04T08:20:24.087 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-06-04T08:20:24.216 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-06-04T08:20:24.288 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:24 +0000 (0:00:05.547) 0:00:57.754 ********* 2020-06-04T08:20:24.298 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-06-04T08:20:24.410 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-06-04T08:20:24.548 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-06-04T08:20:24.733 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-06-04T08:20:24.748 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-06-04T08:20:24.750 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => {"changed": true, "checksum": "3f67a1aec94fcdb4f49608a4758f290e7aa48f9d", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "ca2bb5759342081aa54fc8ea54899674", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1591258824.3151743-232122177841025/source", "state": "file", "uid": 0} 2020-06-04T08:20:24.803 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:24 +0000 (0:00:00.516) 0:00:58.271 ********* 2020-06-04T08:20:24.846 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** ok: [smithi196.front.sepia.ceph.com] => { "msg": "Host smithi196.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-06-04T08:20:24.915 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:24 +0000 (0:00:00.108) 0:00:58.379 ********* 2020-06-04T08:20:24.927 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-04T08:20:24.929 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-06-04T08:20:24.973 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-06-04T08:20:24.975 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-06-04T08:20:25.030 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:25 +0000 (0:00:00.102) 0:00:58.482 ********* TASK [users : Remove revoked users] ******************************************** ok: [smithi060.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-06-04T08:20:25.049 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:25.099 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:25 +0000 (0:00:00.080) 0:00:58.563 ********* 2020-06-04T08:20:25.153 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-06-04T08:20:25.156 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:25.199 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:25 +0000 (0:00:00.103) 0:00:58.667 ********* 2020-06-04T08:20:25.208 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-06-04T08:20:25.210 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-06-04T08:20:25.270 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:25 +0000 (0:00:00.070) 0:00:58.737 ********* 2020-06-04T08:20:25.280 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-06-04T08:20:25.282 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:25.324 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:25 +0000 (0:00:00.054) 0:00:58.792 ********* 2020-06-04T08:20:25.365 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-06-04T08:20:25.367 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:25.402 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:25 +0000 (0:00:00.077) 0:00:58.869 ********* 2020-06-04T08:20:25.447 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:25.490 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:25 +0000 (0:00:00.081) 0:00:58.951 ********* 2020-06-04T08:20:25.519 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:25.558 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:25 +0000 (0:00:00.074) 0:00:59.025 ********* 2020-06-04T08:20:25.604 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:25.651 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:25 +0000 (0:00:00.091) 0:00:59.117 ********* 2020-06-04T08:20:25.690 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-06-04T08:20:25.693 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:25.726 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:25 +0000 (0:00:00.076) 0:00:59.193 ********* 2020-06-04T08:20:25.770 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-06-04T08:20:25.771 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "checksum": "3f67a1aec94fcdb4f49608a4758f290e7aa48f9d", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "ca2bb5759342081aa54fc8ea54899674", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1591258825.3059764-101118197116242/source", "state": "file", "uid": 0} 2020-06-04T08:20:25.824 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:25 +0000 (0:00:00.091) 0:00:59.285 ********* 2020-06-04T08:20:25.829 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-06-04T08:20:25.832 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-04T08:20:25.834 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-04T08:20:25.872 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:25 +0000 (0:00:00.053) 0:00:59.339 ********* 2020-06-04T08:20:25.883 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-06-04T08:20:25.885 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => { "msg": "Host smithi060.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-06-04T08:20:25.937 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:25 +0000 (0:00:00.066) 0:00:59.405 ********* 2020-06-04T08:20:25.946 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-06-04T08:20:25.948 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:25.990 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:25 +0000 (0:00:00.052) 0:00:59.457 ********* 2020-06-04T08:20:26.011 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi060.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-06-04T08:20:26.055 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:26 +0000 (0:00:00.065) 0:00:59.522 ********* 2020-06-04T08:20:26.065 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-06-04T08:20:26.069 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:26.109 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:26 +0000 (0:00:00.054) 0:00:59.577 ********* 2020-06-04T08:20:26.120 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-06-04T08:20:26.122 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:26.166 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:26 +0000 (0:00:00.056) 0:00:59.634 ********* 2020-06-04T08:20:26.176 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-06-04T08:20:26.178 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-04T08:20:26.218 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:26 +0000 (0:00:00.051) 0:00:59.686 ********* 2020-06-04T08:20:26.229 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:26.268 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:26 +0000 (0:00:00.050) 0:00:59.736 ********* 2020-06-04T08:20:26.278 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-06-04T08:20:26.280 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:26.320 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:26 +0000 (0:00:00.051) 0:00:59.787 ********* 2020-06-04T08:20:26.338 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:26.375 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:26 +0000 (0:00:00.055) 0:00:59.843 ********* 2020-06-04T08:20:26.386 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-06-04T08:20:26.388 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-04T08:20:26.426 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:26 +0000 (0:00:00.050) 0:00:59.894 ********* 2020-06-04T08:20:26.437 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-06-04T08:20:26.439 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:26.477 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:26 +0000 (0:00:00.051) 0:00:59.945 ********* 2020-06-04T08:20:26.486 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-06-04T08:20:26.488 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:26.539 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:26 +0000 (0:00:00.061) 0:01:00.006 ********* 2020-06-04T08:20:26.550 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:26.593 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:26 +0000 (0:00:00.054) 0:01:00.061 ********* 2020-06-04T08:20:26.602 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:26.640 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:26 +0000 (0:00:00.046) 0:01:00.108 ********* 2020-06-04T08:20:26.648 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-06-04T08:20:26.651 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:26.688 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:26 +0000 (0:00:00.048) 0:01:00.156 ********* 2020-06-04T08:20:26.699 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-06-04T08:20:26.701 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:26.742 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:26 +0000 (0:00:00.053) 0:01:00.210 ********* 2020-06-04T08:20:26.753 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-06-04T08:20:26.758 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:26.804 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:26 +0000 (0:00:00.060) 0:01:00.270 ********* 2020-06-04T08:20:26.814 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-06-04T08:20:26.822 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:26.852 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:26 +0000 (0:00:00.049) 0:01:00.320 ********* 2020-06-04T08:20:26.864 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-06-04T08:20:26.866 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:26.902 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:26 +0000 (0:00:00.048) 0:01:00.369 ********* 2020-06-04T08:20:26.910 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-06-04T08:20:26.912 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-04T08:20:26.947 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:26 +0000 (0:00:00.046) 0:01:00.415 ********* 2020-06-04T08:20:26.965 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi060.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-04T08:20:26.967 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-04T08:20:27.023 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:27 +0000 (0:00:00.075) 0:01:00.491 ********* 2020-06-04T08:20:27.038 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:27.075 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:27 +0000 (0:00:00.052) 0:01:00.543 ********* 2020-06-04T08:20:27.093 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:27.127 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:27 +0000 (0:00:00.051) 0:01:00.595 ********* 2020-06-04T08:20:27.137 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-06-04T08:20:27.140 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:27.187 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:27 +0000 (0:00:00.059) 0:01:00.655 ********* 2020-06-04T08:20:27.202 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:27.241 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:27 +0000 (0:00:00.053) 0:01:00.708 ********* 2020-06-04T08:20:27.249 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-06-04T08:20:27.250 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:27.294 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:27 +0000 (0:00:00.053) 0:01:00.762 ********* 2020-06-04T08:20:27.307 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-06-04T08:20:27.308 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-04T08:20:27.354 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:27 +0000 (0:00:00.058) 0:01:00.820 ********* 2020-06-04T08:20:27.364 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:27.408 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:27 +0000 (0:00:00.054) 0:01:00.875 ********* 2020-06-04T08:20:27.416 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-06-04T08:20:27.419 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:27.456 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:27 +0000 (0:00:00.048) 0:01:00.924 ********* 2020-06-04T08:20:27.515 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:27.519 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:27 +0000 (0:00:00.062) 0:01:00.986 ********* 2020-06-04T08:20:27.527 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-06-04T08:20:27.529 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-04T08:20:27.566 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:27 +0000 (0:00:00.046) 0:01:01.033 ********* 2020-06-04T08:20:27.574 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-06-04T08:20:27.576 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:27.620 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:27 +0000 (0:00:00.054) 0:01:01.087 ********* 2020-06-04T08:20:27.630 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:27.677 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:27 +0000 (0:00:00.051) 0:01:01.139 ********* 2020-06-04T08:20:27.686 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi196.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-04T08:20:27.729 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:27 +0000 (0:00:00.057) 0:01:01.197 ********* 2020-06-04T08:20:27.741 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-06-04T08:20:27.743 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:27.789 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:27 +0000 (0:00:00.059) 0:01:01.257 ********* 2020-06-04T08:20:27.807 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-06-04T08:20:27.809 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:27.849 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:27 +0000 (0:00:00.059) 0:01:01.316 ********* 2020-06-04T08:20:27.864 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:27.900 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:27 +0000 (0:00:00.050) 0:01:01.367 ********* 2020-06-04T08:20:27.915 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:27.952 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:27 +0000 (0:00:00.052) 0:01:01.420 ********* 2020-06-04T08:20:27.963 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-06-04T08:20:27.965 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:28.020 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:28 +0000 (0:00:00.067) 0:01:01.487 ********* 2020-06-04T08:20:28.028 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-06-04T08:20:28.034 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:28.071 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:28 +0000 (0:00:00.050) 0:01:01.538 ********* 2020-06-04T08:20:28.079 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-06-04T08:20:28.081 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-04T08:20:28.119 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:28 +0000 (0:00:00.048) 0:01:01.586 ********* 2020-06-04T08:20:28.168 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** skipping: [smithi196.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} TASK [common : Clean yum cache] ************************************************ skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:28.201 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:28 +0000 (0:00:00.082) 0:01:01.668 ********* 2020-06-04T08:20:28.245 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:28 +0000 (0:00:00.043) 0:01:01.712 ********* 2020-06-04T08:20:28.254 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-06-04T08:20:28.257 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:28.300 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:28 +0000 (0:00:00.054) 0:01:01.767 ********* 2020-06-04T08:20:28.308 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-06-04T08:20:28.310 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-06-04T08:20:28.362 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:28 +0000 (0:00:00.062) 0:01:01.829 ********* 2020-06-04T08:20:28.372 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-06-04T08:20:28.374 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:28.439 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:28 +0000 (0:00:00.076) 0:01:01.906 ********* 2020-06-04T08:20:28.454 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-06-04T08:20:28.456 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:28.513 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:28 +0000 (0:00:00.073) 0:01:01.980 ********* 2020-06-04T08:20:28.524 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-06-04T08:20:28.526 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:28.571 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:28 +0000 (0:00:00.058) 0:01:02.039 ********* 2020-06-04T08:20:28.614 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:28.668 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:28 +0000 (0:00:00.096) 0:01:02.135 ********* 2020-06-04T08:20:28.703 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-06-04T08:20:28.705 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.015947", "end": "2020-06-04 08:20:28.627626", "rc": 0, "start": "2020-06-04 08:20:28.611679", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-06-04T08:20:28.741 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:28 +0000 (0:00:00.073) 0:01:02.208 ********* 2020-06-04T08:20:28.753 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-06-04T08:20:28.754 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:28.804 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:28 +0000 (0:00:00.063) 0:01:02.272 ********* 2020-06-04T08:20:28.814 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-06-04T08:20:28.815 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:28.856 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:28 +0000 (0:00:00.051) 0:01:02.323 ********* 2020-06-04T08:20:28.865 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi060.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-04T08:20:28.905 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:28 +0000 (0:00:00.049) 0:01:02.372 ********* 2020-06-04T08:20:28.917 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:28.959 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:28 +0000 (0:00:00.050) 0:01:02.423 ********* 2020-06-04T08:20:28.965 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:29.018 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:29 +0000 (0:00:00.061) 0:01:02.485 ********* 2020-06-04T08:20:29.041 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-06-04T08:20:29.045 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:29.098 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:29 +0000 (0:00:00.080) 0:01:02.565 ********* 2020-06-04T08:20:29.110 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-06-04T08:20:29.116 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:29.162 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:29 +0000 (0:00:00.064) 0:01:02.630 ********* 2020-06-04T08:20:29.174 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:29.213 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:29 +0000 (0:00:00.050) 0:01:02.681 ********* 2020-06-04T08:20:29.224 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-06-04T08:20:29.230 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:29.272 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:29 +0000 (0:00:00.058) 0:01:02.739 ********* 2020-06-04T08:20:29.291 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-06-04T08:20:29.293 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:29.346 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:29 +0000 (0:00:00.069) 0:01:02.809 ********* 2020-06-04T08:20:29.351 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-06-04T08:20:29.352 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:29.354 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:29.396 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:29 +0000 (0:00:00.054) 0:01:02.864 ********* 2020-06-04T08:20:29.409 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-06-04T08:20:29.411 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:29.476 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:29 +0000 (0:00:00.079) 0:01:02.944 ********* 2020-06-04T08:20:29.487 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-06-04T08:20:29.489 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-06-04T08:20:29.535 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:29 +0000 (0:00:00.059) 0:01:03.003 ********* 2020-06-04T08:20:29.547 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-06-04T08:20:29.549 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:29.593 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:29 +0000 (0:00:00.054) 0:01:03.058 ********* 2020-06-04T08:20:29.601 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-06-04T08:20:29.606 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:29.660 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:29 +0000 (0:00:00.069) 0:01:03.127 ********* 2020-06-04T08:20:29.828 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-06-04T08:20:29.831 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.012694", "end": "2020-06-04 08:20:30.214029", "rc": 0, "start": "2020-06-04 08:20:30.201335", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-06-04T08:20:29.881 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:29 +0000 (0:00:00.221) 0:01:03.349 ********* 2020-06-04T08:20:29.921 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-06-04T08:20:29.923 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:29.983 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:29 +0000 (0:00:00.101) 0:01:03.450 ********* 2020-06-04T08:20:30.013 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:30.063 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:30 +0000 (0:00:00.080) 0:01:03.531 ********* 2020-06-04T08:20:30.107 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-06-04T08:20:30.109 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:30.164 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:30 +0000 (0:00:00.101) 0:01:03.632 ********* 2020-06-04T08:20:30.198 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:30.245 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:30 +0000 (0:00:00.080) 0:01:03.712 ********* 2020-06-04T08:20:30.299 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-06-04T08:20:30.301 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:30.345 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:30 +0000 (0:00:00.100) 0:01:03.813 ********* 2020-06-04T08:20:30.384 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:30.440 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:30 +0000 (0:00:00.095) 0:01:03.908 ********* 2020-06-04T08:20:30.492 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-06-04T08:20:30.494 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:30.529 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:30 +0000 (0:00:00.088) 0:01:03.996 ********* 2020-06-04T08:20:30.584 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-06-04T08:20:30.586 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.843542", "end": "2020-06-04 08:20:30.500336", "rc": 0, "start": "2020-06-04 08:20:29.656794", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-04T08:20:30.637 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:30 +0000 (0:00:00.108) 0:01:04.105 ********* 2020-06-04T08:20:30.683 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-06-04T08:20:30.685 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:30.721 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:30 +0000 (0:00:00.083) 0:01:04.188 ********* 2020-06-04T08:20:30.899 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ [WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi196.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.023695", "end": "2020-06-04 08:20:30.816510", "rc": 0, "start": "2020-06-04 08:20:30.792815", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-04T08:20:30.942 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:30 +0000 (0:00:00.218) 0:01:04.407 ********* 2020-06-04T08:20:31.115 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.427914", "end": "2020-06-04 08:20:31.500384", "rc": 0, "start": "2020-06-04 08:20:31.072470", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-04T08:20:31.173 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:31 +0000 (0:00:00.232) 0:01:04.640 ********* 2020-06-04T08:20:31.214 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:31.258 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:31 +0000 (0:00:00.085) 0:01:04.725 ********* 2020-06-04T08:20:31.422 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-06-04T08:20:31.424 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.017738", "end": "2020-06-04 08:20:31.810141", "rc": 0, "start": "2020-06-04 08:20:31.792403", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-04T08:20:31.462 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:31 +0000 (0:00:00.201) 0:01:04.927 ********* 2020-06-04T08:20:35.658 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** [WARNING]: Could not find aptitude. Using apt-get instead changed: [smithi196.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1591258833, "cache_updated": true, "changed": true} 2020-06-04T08:20:35.703 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:35 +0000 (0:00:04.243) 0:01:09.171 ********* 2020-06-04T08:20:36.215 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1591258834, "cache_updated": true, "changed": true} 2020-06-04T08:20:36.266 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:36 +0000 (0:00:00.563) 0:01:09.734 ********* 2020-06-04T08:20:36.569 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-06-04T08:20:36.571 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"cache_update_time": 1591258833, "cache_updated": false, "changed": false} 2020-06-04T08:20:36.613 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:36 +0000 (0:00:00.343) 0:01:10.077 ********* 2020-06-04T08:20:36.642 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-06-04T08:20:36.645 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:36.687 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:36 +0000 (0:00:00.077) 0:01:10.155 ********* 2020-06-04T08:20:37.251 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-06-04T08:20:37.255 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1591258834, "cache_updated": false, "changed": false} 2020-06-04T08:20:37.311 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:37 +0000 (0:00:00.623) 0:01:10.779 ********* 2020-06-04T08:20:37.358 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-06-04T08:20:37.360 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:37.395 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:37 +0000 (0:00:00.083) 0:01:10.863 ********* 2020-06-04T08:20:37.408 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-06-04T08:20:37.410 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-06-04T08:20:37.456 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:37 +0000 (0:00:00.060) 0:01:10.923 ********* 2020-06-04T08:20:37.782 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-06-04T08:20:37.821 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:37 +0000 (0:00:00.364) 0:01:11.288 ********* 2020-06-04T08:20:38.335 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** ok: [smithi196.front.sepia.ceph.com] => {"cache_update_time": 1591258833, "cache_updated": false, "changed": false} 2020-06-04T08:20:38.397 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:38 +0000 (0:00:00.576) 0:01:11.864 ********* 2020-06-04T08:20:38.758 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1591258834, "cache_updated": false, "changed": false} 2020-06-04T08:20:38.795 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:38 +0000 (0:00:00.398) 0:01:12.262 ********* 2020-06-04T08:20:38.814 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-06-04T08:20:38.816 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-06-04T08:20:39.225 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-06-04T08:20:39.236 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-06-04T08:20:39.578 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-06-04T08:20:39.603 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-06-04T08:20:39.662 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:39 +0000 (0:00:00.864) 0:01:13.127 ********* 2020-06-04T08:20:39.830 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-06-04T08:20:39.832 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-06-04T08:20:39.870 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:39 +0000 (0:00:00.210) 0:01:13.337 ********* 2020-06-04T08:20:39.946 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-06-04T08:20:39.954 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-06-04T08:20:40.006 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:40 +0000 (0:00:00.136) 0:01:13.474 ********* 2020-06-04T08:20:40.154 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-06-04T08:20:40.200 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:40 +0000 (0:00:00.194) 0:01:13.668 ********* 2020-06-04T08:20:40.240 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-06-04T08:20:40.242 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-06-04T08:20:40.602 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} ok: [smithi196.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-06-04T08:20:40.931 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-06-04T08:20:40.932 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-06-04T08:20:40.973 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:40 +0000 (0:00:00.772) 0:01:14.440 ********* 2020-06-04T08:20:41.017 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-06-04T08:20:41.018 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-06-04T08:20:41.056 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:41 +0000 (0:00:00.082) 0:01:14.523 ********* 2020-06-04T08:20:41.106 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:41.147 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:41 +0000 (0:00:00.091) 0:01:14.615 ********* 2020-06-04T08:20:41.186 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-06-04T08:20:41.190 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-04T08:20:41.226 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:41 +0000 (0:00:00.078) 0:01:14.693 ********* 2020-06-04T08:20:41.287 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [smithi196.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi060.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-06-04T08:20:41.290 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [smithi196.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-06-04T08:20:41.342 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:41 +0000 (0:00:00.116) 0:01:14.810 ********* 2020-06-04T08:20:41.352 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-06-04T08:20:41.354 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-06-04T08:20:41.397 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:41 +0000 (0:00:00.048) 0:01:14.858 ********* 2020-06-04T08:20:41.399 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-06-04T08:20:41.404 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-06-04T08:20:41.438 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:41 +0000 (0:00:00.047) 0:01:14.906 ********* 2020-06-04T08:20:41.458 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:41.494 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:41 +0000 (0:00:00.053) 0:01:14.960 ********* 2020-06-04T08:20:41.502 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:41.554 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:41 +0000 (0:00:00.060) 0:01:15.020 ********* 2020-06-04T08:20:41.595 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* skipping: [smithi060.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-04T08:20:41.641 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:41 +0000 (0:00:00.087) 0:01:15.108 ********* 2020-06-04T08:20:41.694 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [smithi060.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-06-04T08:20:41.705 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-06-04T08:20:41.717 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-06-04T08:20:41.758 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:41 +0000 (0:00:00.115) 0:01:15.224 ********* 2020-06-04T08:20:41.812 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:41.848 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:41 +0000 (0:00:00.091) 0:01:15.315 ********* 2020-06-04T08:20:44.594 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** changed: [smithi196.front.sepia.ceph.com] => {"cache_update_time": 1591258833, "cache_updated": false, "changed": true, "stderr": "", "stderr_lines": [], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages will be upgraded:\n nagios-plugins-basic\n1 upgraded, 0 newly installed, 0 to remove and 54 not upgraded.\nNeed to get 6444 B of archives.\nAfter this operation, 0 B of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 nagios-plugins-basic all 2.2-3ubuntu3 [6444 B]\nFetched 6444 B in 0s (34.5 kB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137516 files and directories currently installed.)\r\nPreparing to unpack .../nagios-plugins-basic_2.2-3ubuntu3_all.deb ...\r\nUnpacking nagios-plugins-basic (2.2-3ubuntu3) over (2.2-3ubuntu2) ...\r\nSetting up nagios-plugins-basic (2.2-3ubuntu3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages will be upgraded:", " nagios-plugins-basic", "1 upgraded, 0 newly installed, 0 to remove and 54 not upgraded.", "Need to get 6444 B of archives.", "After this operation, 0 B of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 nagios-plugins-basic all 2020-06-04T08:20:44.596 INFO:teuthology.task.ansible.out: 2.2-3ubuntu3 [6444 B]", "Fetched 6444 B in 0s (34.5 kB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137516 files and directories currently installed.)", "Preparing to unpack .../nagios-plugins-basic_2.2-3ubuntu3_all.deb ...", "Unpacking nagios-plugins-basic (2.2-3ubuntu3) over (2.2-3ubuntu2) ...", "Setting up nagios-plugins-basic (2.2-3ubuntu3) ..."]} 2020-06-04T08:20:44.640 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:44 +0000 (0:00:02.791) 0:01:18.107 ********* 2020-06-04T08:20:45.036 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-06-04T08:20:45.038 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-06-04T08:20:45.086 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:45 +0000 (0:00:00.443) 0:01:18.551 ********* 2020-06-04T08:20:45.094 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-06-04T08:20:45.096 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1591258834, "cache_updated": false, "changed": true, "stderr": "", "stderr_lines": [], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages will be upgraded:\n nagios-plugins-basic\n1 upgraded, 0 newly installed, 0 to remove and 54 not upgraded.\nNeed to get 6444 B of archives.\nAfter this operation, 0 B of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 nagios-plugins-basic all 2.2-3ubuntu3 [6444 B]\nFetched 6444 B in 0s (39.0 kB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137516 files and directories currently installed.)\r\nPreparing to unpack .../nagios-plugins-basic_2.2-3ubuntu3_all.deb ...\r\nUnpacking nagios-plugins-basic (2.2-3ubuntu3) over (2.2-3ubuntu2) ...\r\nSetting up nagios-plugins-basic (2.2-3ubuntu3) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages will be upgraded:", " nagios-plugins-basic", "1 upgraded, 0 newly installed, 0 to remove and 54 not upgraded.", "Need to get 6444 B of archives.", "After this operation, 0 B of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 nagios-plugins-basic all 2.2-3ubuntu3 [6444 B]", "Fetched 6444 B in 0s (39.0 kB/s)", "(Reading database ... 2020-06-04T08:20:45.099 INFO:teuthology.task.ansible.out:", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137516 files and directories currently installed.)", "Preparing to unpack .../nagios-plugins-basic_2.2-3ubuntu3_all.deb ...", "Unpacking nagios-plugins-basic (2.2-3ubuntu3) over (2.2-3ubuntu2) ...", "Setting up nagios-plugins-basic (2.2-3ubuntu3) ..."]} 2020-06-04T08:20:45.137 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:45 +0000 (0:00:00.049) 0:01:18.601 ********* 2020-06-04T08:20:45.459 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-06-04T08:20:45.473 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-06-04T08:20:45.517 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:45 +0000 (0:00:00.378) 0:01:18.980 ********* 2020-06-04T08:20:45.532 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-06-04T08:20:45.534 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2020-06-04T08:20:45.579 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:45 +0000 (0:00:00.066) 0:01:19.047 ********* 2020-06-04T08:20:45.694 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-06-04T08:20:45.696 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-06-04T08:20:45.846 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-06-04T08:20:45.894 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:45 +0000 (0:00:00.313) 0:01:19.360 ********* 2020-06-04T08:20:45.929 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:45.965 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:45 +0000 (0:00:00.071) 0:01:19.432 ********* 2020-06-04T08:20:45.974 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-06-04T08:20:45.975 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2020-06-04T08:20:46.022 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:46 +0000 (0:00:00.056) 0:01:19.489 ********* 2020-06-04T08:20:46.032 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** skipping: [smithi196.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-04T08:20:46.069 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:46 +0000 (0:00:00.047) 0:01:19.536 ********* 2020-06-04T08:20:46.107 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-06-04T08:20:46.109 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:46.152 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:46 +0000 (0:00:00.083) 0:01:19.619 ********* 2020-06-04T08:20:46.195 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ ok: [smithi060.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2020-06-04T08:20:46.321 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2020-06-04T08:20:46.363 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:46 +0000 (0:00:00.211) 0:01:19.831 ********* 2020-06-04T08:20:46.407 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-06-04T08:20:46.410 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:46.454 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:46 +0000 (0:00:00.090) 0:01:19.921 ********* 2020-06-04T08:20:46.497 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-06-04T08:20:46.499 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-06-04T08:20:46.562 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:46 +0000 (0:00:00.107) 0:01:20.029 ********* 2020-06-04T08:20:46.603 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:46.647 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:46 +0000 (0:00:00.085) 0:01:20.114 ********* 2020-06-04T08:20:46.658 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-06-04T08:20:46.662 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "checksum": "9c3d1d79583cf76a6d81bb2ec150bc4af5ca3c5c", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1166, "state": "file", "uid": 0} 2020-06-04T08:20:46.700 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:46 +0000 (0:00:00.053) 0:01:20.168 ********* 2020-06-04T08:20:47.072 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "9c3d1d79583cf76a6d81bb2ec150bc4af5ca3c5c", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1166, "state": "file", "uid": 0} 2020-06-04T08:20:47.115 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:47 +0000 (0:00:00.415) 0:01:20.583 ********* 2020-06-04T08:20:47.462 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-06-04T08:20:47.466 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-06-04 08:18:50 UTC", "ActiveEnterTimestampMonotonic": "19463930", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "remote-fs.target local-fs.target basic.target systemd-tmpfiles-setup.service system.slice time-sync.target systemd-journald.socket nss-lookup.target -.mount sysinit.target var-run.mount network.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-06-04 08:18:50 UTC", "AssertTimestampMonotonic": "19462464", "Before": "getty@tty1.service xdm.service plymouth-quit.service multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-06-04 08:18:50 UTC", "ConditionTimestampMonotonic": "19462464", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2020-06-04T08:20:47.478 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1263", "ExecMainStartTimestamp": "Thu 2020-06-04 08:18:50 UTC", "ExecMainStartTimestampMonotonic": "19463871", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2020-06-04 08:18:50 UTC] ; stop_time=[n/a] ; pid=1263 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-06-04 08:18:50 UTC", "InactiveExitTimestampMonotonic": "19463930", "InvocationID": "91211266c73343d6b9a85d222b6fc467", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2020-06-04T08:20:47.482 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1263", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2020-06-04T08:20:47.502 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-06-04 08:18:50 UTC", "StateChangeTimestampMonotonic": "19463930", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-06-04 08:18:50 UTC", "WatchdogTimestampMonotonic": "19463928", "WatchdogUSec": "0"}} ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-06-04 08:19:09 UTC", "ActiveEnterTimestampMonotonic": "18240149", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "time-sync.target systemd-journald.socket sysinit.target var-run.mount local-fs.target -.mount remote-fs.target network.target nss-lookup.target 2020-06-04T08:20:47.507 INFO:teuthology.task.ansible.out: systemd-tmpfiles-setup.service basic.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-06-04 08:19:09 UTC", "AssertTimestampMonotonic": "18239305", "Before": "getty@tty1.service plymouth-quit.service multi-user.target xdm.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-06-04 08:19:09 UTC", "ConditionTimestampMonotonic": "18239305", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-server.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1166", "ExecMainStartTimestamp": "Thu 2020-06-04 08:19:09 UTC", "ExecMai 2020-06-04T08:20:47.514 INFO:teuthology.task.ansible.out:nStartTimestampMonotonic": "18240113", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Thu 2020-06-04 08:19:09 UTC] ; stop_time=[n/a] ; pid=1166 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-06-04 08:19:09 UTC", "InactiveExitTimestampMonotonic": "18240149", "InvocationID": "c7458cc7142e4391ada220e038447e88", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft 2020-06-04T08:20:47.526 INFO:teuthology.task.ansible.out:": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1166", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "St 2020-06-04T08:20:47.529 INFO:teuthology.task.ansible.out:artLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-06-04 08:19:09 UTC", "StateChangeTimestampMonotonic": "18240149", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-06-04 08:19:09 UTC", "WatchdogTimestampMonotonic": "18240148", "WatchdogUSec": "0"}} Thursday 04 June 2020 08:20:47 +0000 (0:00:00.414) 0:01:20.997 ********* 2020-06-04T08:20:47.574 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:47 +0000 (0:00:00.040) 0:01:21.037 ********* 2020-06-04T08:20:47.585 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:47.626 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:47 +0000 (0:00:00.054) 0:01:21.092 ********* 2020-06-04T08:20:47.638 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:47.674 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:47 +0000 (0:00:00.048) 0:01:21.140 ********* 2020-06-04T08:20:47.684 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-06-04T08:20:47.686 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:47.723 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:47 +0000 (0:00:00.049) 0:01:21.190 ********* 2020-06-04T08:20:47.737 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:47.775 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:47 +0000 (0:00:00.051) 0:01:21.242 ********* 2020-06-04T08:20:47.784 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:47.827 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:47 +0000 (0:00:00.051) 0:01:21.294 ********* 2020-06-04T08:20:47.838 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:47.886 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:47 +0000 (0:00:00.059) 0:01:21.353 ********* 2020-06-04T08:20:47.898 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:47.942 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:47 +0000 (0:00:00.055) 0:01:21.409 ********* 2020-06-04T08:20:47.955 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:48.015 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:48 +0000 (0:00:00.071) 0:01:21.481 ********* 2020-06-04T08:20:48.022 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-06-04T08:20:48.024 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:48.067 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:48 +0000 (0:00:00.052) 0:01:21.534 ********* 2020-06-04T08:20:48.077 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:48.116 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:48 +0000 (0:00:00.049) 0:01:21.584 ********* 2020-06-04T08:20:48.125 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-06-04T08:20:48.127 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:48.167 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:48 +0000 (0:00:00.050) 0:01:21.634 ********* 2020-06-04T08:20:48.178 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:48.217 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:48 +0000 (0:00:00.049) 0:01:21.684 ********* 2020-06-04T08:20:48.228 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-06-04T08:20:48.230 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:48.289 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:48 +0000 (0:00:00.071) 0:01:21.756 ********* 2020-06-04T08:20:48.300 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:48.345 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:48 +0000 (0:00:00.056) 0:01:21.813 ********* 2020-06-04T08:20:48.357 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:48.396 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:48 +0000 (0:00:00.050) 0:01:21.863 ********* 2020-06-04T08:20:48.405 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:48.445 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:48 +0000 (0:00:00.049) 0:01:21.913 ********* 2020-06-04T08:20:48.456 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-06-04T08:20:48.463 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:48.500 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:48 +0000 (0:00:00.055) 0:01:21.968 ********* 2020-06-04T08:20:48.510 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:48.555 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:48 +0000 (0:00:00.053) 0:01:22.022 ********* 2020-06-04T08:20:48.570 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:48.602 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:48 +0000 (0:00:00.048) 0:01:22.070 ********* 2020-06-04T08:20:48.611 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:48.653 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:48 +0000 (0:00:00.050) 0:01:22.120 ********* 2020-06-04T08:20:48.672 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-06-04T08:20:48.674 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-06-04T08:20:48.719 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:48 +0000 (0:00:00.066) 0:01:22.187 ********* 2020-06-04T08:20:48.733 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2020-06-04T08:20:48.780 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:48 +0000 (0:00:00.061) 0:01:22.248 ********* 2020-06-04T08:20:48.793 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-06-04T08:20:48.797 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-06-04T08:20:48.839 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:48 +0000 (0:00:00.058) 0:01:22.307 ********* 2020-06-04T08:20:48.859 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-06-04T08:20:48.907 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:48 +0000 (0:00:00.067) 0:01:22.374 ********* 2020-06-04T08:20:48.917 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** ok: [smithi060.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-06-04T08:20:48.966 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:48 +0000 (0:00:00.053) 0:01:22.428 ********* 2020-06-04T08:20:48.979 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-06-04T08:20:49.026 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:49 +0000 (0:00:00.065) 0:01:22.493 ********* 2020-06-04T08:20:49.089 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ ok: [smithi060.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} Thursday 04 June 2020 08:20:49 +0000 (0:00:00.055) 0:01:22.548 ********* 2020-06-04T08:20:49.094 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-06-04T08:20:49.140 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:49 +0000 (0:00:00.058) 0:01:22.607 ********* 2020-06-04T08:20:49.241 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-06-04T08:20:49.287 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:49 +0000 (0:00:00.146) 0:01:22.754 ********* 2020-06-04T08:20:49.347 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-06-04T08:20:49.390 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:49 +0000 (0:00:00.098) 0:01:22.853 ********* 2020-06-04T08:20:49.491 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "name": "ubuntu", "state": "present", "system": false} 2020-06-04T08:20:49.553 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:49 +0000 (0:00:00.162) 0:01:23.015 ********* 2020-06-04T08:20:49.563 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "name": "ubuntu", "state": "present", "system": false} 2020-06-04T08:20:49.597 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:49 +0000 (0:00:00.049) 0:01:23.064 ********* 2020-06-04T08:20:49.606 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-06-04T08:20:49.608 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:49.648 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:49 +0000 (0:00:00.051) 0:01:23.115 ********* 2020-06-04T08:20:49.658 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:49.703 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:49 +0000 (0:00:00.054) 0:01:23.170 ********* 2020-06-04T08:20:49.712 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-06-04T08:20:49.713 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:49.751 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:49 +0000 (0:00:00.047) 0:01:23.218 ********* 2020-06-04T08:20:49.761 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:49.802 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:49 +0000 (0:00:00.051) 0:01:23.269 ********* 2020-06-04T08:20:49.937 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi060.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-06-04T08:20:49.975 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:49 +0000 (0:00:00.173) 0:01:23.443 ********* 2020-06-04T08:20:49.986 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-06-04T08:20:50.026 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:50 +0000 (0:00:00.050) 0:01:23.493 ********* 2020-06-04T08:20:50.467 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-06-04T08:20:50.471 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-06-04T08:20:50.526 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:50 +0000 (0:00:00.497) 0:01:23.991 ********* 2020-06-04T08:20:50.532 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-06-04T08:20:50.579 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:50 +0000 (0:00:00.055) 0:01:24.047 ********* 2020-06-04T08:20:50.606 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi060"}, "changed": false} 2020-06-04T08:20:50.629 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:50 +0000 (0:00:00.050) 0:01:24.097 ********* 2020-06-04T08:20:50.639 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi196"}, "changed": false} 2020-06-04T08:20:50.684 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:50 +0000 (0:00:00.054) 0:01:24.152 ********* 2020-06-04T08:20:51.661 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-06-04T08:20:51.664 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi060.front.sepia.ceph.com", "ansible_hostname": "smithi060", "ansible_nodename": "smithi060"}, "changed": true, "name": "smithi060"} 2020-06-04T08:20:51.703 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:51 +0000 (0:00:01.018) 0:01:25.170 ********* 2020-06-04T08:20:51.714 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi196.front.sepia.ceph.com", "ansible_hostname": "smithi196", "ansible_nodename": "smithi196"}, "changed": true, "name": "smithi196"} 2020-06-04T08:20:51.756 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:51 +0000 (0:00:00.053) 0:01:25.223 ********* 2020-06-04T08:20:52.104 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-06-04T08:20:52.107 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-06-04T08:20:52.157 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:52 +0000 (0:00:00.393) 0:01:25.617 ********* 2020-06-04T08:20:52.172 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-06-04T08:20:52.228 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:52 +0000 (0:00:00.078) 0:01:25.696 ********* 2020-06-04T08:20:52.563 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-06-04T08:20:52.608 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:52 +0000 (0:00:00.379) 0:01:26.076 ********* 2020-06-04T08:20:52.652 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-06-04T08:20:52.692 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:52 +0000 (0:00:00.084) 0:01:26.160 ********* 2020-06-04T08:20:53.177 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-06-04T08:20:53.179 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-06-04T08:20:53.230 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:53 +0000 (0:00:00.538) 0:01:26.698 ********* 2020-06-04T08:20:53.242 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-06-04T08:20:53.285 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:53 +0000 (0:00:00.054) 0:01:26.752 ********* 2020-06-04T08:20:53.295 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-06-04T08:20:53.302 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:53.341 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:53 +0000 (0:00:00.056) 0:01:26.808 ********* 2020-06-04T08:20:53.352 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:53.399 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:53 +0000 (0:00:00.058) 0:01:26.867 ********* 2020-06-04T08:20:53.410 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-06-04T08:20:53.412 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:53.468 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:53 +0000 (0:00:00.068) 0:01:26.936 ********* 2020-06-04T08:20:53.483 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:53.530 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:53 +0000 (0:00:00.062) 0:01:26.998 ********* 2020-06-04T08:20:53.542 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-06-04T08:20:53.544 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:53.603 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:53 +0000 (0:00:00.072) 0:01:27.071 ********* 2020-06-04T08:20:53.616 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:53.656 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:53 +0000 (0:00:00.053) 0:01:27.124 ********* 2020-06-04T08:20:53.667 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-06-04T08:20:53.669 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:53.733 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:53 +0000 (0:00:00.065) 0:01:27.189 ********* skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:53.781 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:53 +0000 (0:00:00.053) 0:01:27.243 ********* 2020-06-04T08:20:53.786 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-06-04T08:20:53.788 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:53.825 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:53 +0000 (0:00:00.049) 0:01:27.293 ********* 2020-06-04T08:20:53.836 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:53.877 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:53 +0000 (0:00:00.051) 0:01:27.345 ********* 2020-06-04T08:20:53.889 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-06-04T08:20:53.891 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:53.928 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:53 +0000 (0:00:00.050) 0:01:27.395 ********* 2020-06-04T08:20:53.938 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:53.986 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:53 +0000 (0:00:00.057) 0:01:27.453 ********* 2020-06-04T08:20:53.994 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-06-04T08:20:53.996 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:54.044 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:54 +0000 (0:00:00.057) 0:01:27.511 ********* 2020-06-04T08:20:54.053 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:54.100 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:54 +0000 (0:00:00.055) 0:01:27.567 ********* 2020-06-04T08:20:54.108 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-06-04T08:20:54.109 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:54.150 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:54 +0000 (0:00:00.048) 0:01:27.615 ********* 2020-06-04T08:20:54.162 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:54.216 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:54 +0000 (0:00:00.067) 0:01:27.683 ********* 2020-06-04T08:20:54.229 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-06-04T08:20:54.266 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:54 +0000 (0:00:00.050) 0:01:27.734 ********* 2020-06-04T08:20:54.317 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:54 +0000 (0:00:00.050) 0:01:27.784 ********* 2020-06-04T08:20:54.346 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-06-04T08:20:54.384 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:54 +0000 (0:00:00.058) 0:01:27.843 ********* 2020-06-04T08:20:54.433 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:54 +0000 (0:00:00.057) 0:01:27.900 ********* 2020-06-04T08:20:54.444 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-06-04T08:20:54.481 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:54 +0000 (0:00:00.047) 0:01:27.948 ********* 2020-06-04T08:20:54.534 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:54 +0000 (0:00:00.049) 0:01:27.998 ********* 2020-06-04T08:20:54.539 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-06-04T08:20:54.540 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:54.582 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:54 +0000 (0:00:00.046) 0:01:28.045 ********* 2020-06-04T08:20:54.598 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:54.640 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:54 +0000 (0:00:00.062) 0:01:28.108 ********* 2020-06-04T08:20:54.651 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-06-04T08:20:54.653 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-06-04T08:20:54.701 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:54 +0000 (0:00:00.060) 0:01:28.168 ********* 2020-06-04T08:20:54.717 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-06-04T08:20:54.719 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-06-04T08:20:54.768 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:54 +0000 (0:00:00.067) 0:01:28.236 ********* 2020-06-04T08:20:54.783 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-06-04T08:20:54.786 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:54.834 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:54 +0000 (0:00:00.066) 0:01:28.302 ********* 2020-06-04T08:20:54.847 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:54.906 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:54 +0000 (0:00:00.069) 0:01:28.371 ********* 2020-06-04T08:20:54.913 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-04T08:20:54.915 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:54.961 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:54 +0000 (0:00:00.057) 0:01:28.429 ********* 2020-06-04T08:20:54.970 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:55.019 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:55 +0000 (0:00:00.057) 0:01:28.486 ********* 2020-06-04T08:20:55.038 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:55.066 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:55 +0000 (0:00:00.047) 0:01:28.534 ********* 2020-06-04T08:20:55.075 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:55.117 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:55 +0000 (0:00:00.049) 0:01:28.584 ********* 2020-06-04T08:20:55.126 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-06-04T08:20:55.127 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:55.163 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:55 +0000 (0:00:00.046) 0:01:28.631 ********* 2020-06-04T08:20:55.172 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:55.216 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:55 +0000 (0:00:00.053) 0:01:28.684 ********* 2020-06-04T08:20:55.242 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:55.265 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:55 +0000 (0:00:00.049) 0:01:28.733 ********* 2020-06-04T08:20:55.275 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:55.316 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:55 +0000 (0:00:00.050) 0:01:28.783 ********* 2020-06-04T08:20:55.326 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-06-04T08:20:55.328 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:55.367 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:55 +0000 (0:00:00.051) 0:01:28.835 ********* 2020-06-04T08:20:55.387 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:55.421 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:55 +0000 (0:00:00.053) 0:01:28.888 ********* 2020-06-04T08:20:55.434 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:55.468 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:55 +0000 (0:00:00.047) 0:01:28.936 ********* 2020-06-04T08:20:55.479 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:55.531 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:55 +0000 (0:00:00.056) 0:01:28.993 ********* 2020-06-04T08:20:55.535 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-06-04T08:20:55.537 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:55.575 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:55 +0000 (0:00:00.049) 0:01:29.042 ********* 2020-06-04T08:20:55.584 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:55.630 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:55 +0000 (0:00:00.053) 0:01:29.096 ********* 2020-06-04T08:20:55.638 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-06-04T08:20:55.646 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:55.691 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:55 +0000 (0:00:00.062) 0:01:29.159 ********* 2020-06-04T08:20:55.701 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:55.742 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:55 +0000 (0:00:00.050) 0:01:29.210 ********* 2020-06-04T08:20:55.753 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-06-04T08:20:55.755 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:55.796 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:55 +0000 (0:00:00.052) 0:01:29.263 ********* 2020-06-04T08:20:55.805 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:55.844 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:55 +0000 (0:00:00.049) 0:01:29.312 ********* 2020-06-04T08:20:55.854 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-06-04T08:20:55.855 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:55.894 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:55 +0000 (0:00:00.049) 0:01:29.362 ********* 2020-06-04T08:20:55.910 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:55.943 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:55 +0000 (0:00:00.048) 0:01:29.410 ********* 2020-06-04T08:20:55.952 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-06-04T08:20:55.954 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:55.992 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:55 +0000 (0:00:00.049) 0:01:29.460 ********* 2020-06-04T08:20:56.006 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:56.053 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:56 +0000 (0:00:00.060) 0:01:29.520 ********* 2020-06-04T08:20:56.062 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-06-04T08:20:56.064 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:56.101 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:56 +0000 (0:00:00.047) 0:01:29.568 ********* 2020-06-04T08:20:56.110 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:56.149 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:56 +0000 (0:00:00.048) 0:01:29.617 ********* 2020-06-04T08:20:56.159 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-06-04T08:20:56.162 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:56.200 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:56 +0000 (0:00:00.050) 0:01:29.668 ********* 2020-06-04T08:20:56.212 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:56.255 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:56 +0000 (0:00:00.054) 0:01:29.722 ********* 2020-06-04T08:20:56.264 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-06-04T08:20:56.266 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:56.305 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:56 +0000 (0:00:00.050) 0:01:29.772 ********* 2020-06-04T08:20:56.320 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:56.364 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:56 +0000 (0:00:00.059) 0:01:29.832 ********* 2020-06-04T08:20:56.378 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:56.414 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:56 +0000 (0:00:00.049) 0:01:29.881 ********* 2020-06-04T08:20:56.426 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:56.469 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:56 +0000 (0:00:00.055) 0:01:29.937 ********* 2020-06-04T08:20:56.480 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-06-04T08:20:56.481 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:56.520 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:56 +0000 (0:00:00.050) 0:01:29.988 ********* 2020-06-04T08:20:56.532 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:56.574 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:56 +0000 (0:00:00.053) 0:01:30.042 ********* 2020-06-04T08:20:56.585 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-06-04T08:20:56.590 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:56.625 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:56 +0000 (0:00:00.051) 0:01:30.093 ********* 2020-06-04T08:20:56.641 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:56.680 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:56 +0000 (0:00:00.053) 0:01:30.147 ********* 2020-06-04T08:20:56.688 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2020-06-04T08:20:56.707 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:56.728 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:56 +0000 (0:00:00.049) 0:01:30.196 ********* 2020-06-04T08:20:56.737 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:56.782 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:56 +0000 (0:00:00.051) 0:01:30.248 ********* 2020-06-04T08:20:56.788 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-06-04T08:20:56.790 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:56.827 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:56 +0000 (0:00:00.046) 0:01:30.294 ********* 2020-06-04T08:20:56.836 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:56.872 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:56 +0000 (0:00:00.045) 0:01:30.340 ********* 2020-06-04T08:20:56.998 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-06-04T08:20:57.000 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.004489", "end": "2020-06-04 08:20:57.353469", "rc": 0, "start": "2020-06-04 08:20:57.348980", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-04T08:20:57.054 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:57 +0000 (0:00:00.175) 0:01:30.516 ********* 2020-06-04T08:20:57.062 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005135", "end": "2020-06-04 08:20:56.981913", "rc": 0, "start": "2020-06-04 08:20:56.976778", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-04T08:20:57.107 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:57 +0000 (0:00:00.059) 0:01:30.575 ********* 2020-06-04T08:20:57.121 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-06-04T08:20:57.169 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:57 +0000 (0:00:00.061) 0:01:30.636 ********* 2020-06-04T08:20:57.219 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:57 +0000 (0:00:00.050) 0:01:30.687 ********* 2020-06-04T08:20:57.585 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-06-04T08:20:57.671 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:57 +0000 (0:00:00.451) 0:01:31.138 ********* 2020-06-04T08:20:57.683 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-06-04T08:20:57.746 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:57 +0000 (0:00:00.071) 0:01:31.210 ********* 2020-06-04T08:20:57.752 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-06-04T08:20:57.754 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:57.802 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:57 +0000 (0:00:00.054) 0:01:31.264 ********* 2020-06-04T08:20:57.813 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:20:57.871 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:20:57 +0000 (0:00:00.071) 0:01:31.335 ********* 2020-06-04T08:20:59.182 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* ok: [smithi196.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-06-04T08:20:59.236 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-06-04T08:21:00.130 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-06-04T08:21:00.165 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:00 +0000 (0:00:02.296) 0:01:33.632 ********* 2020-06-04T08:21:00.228 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-06-04T08:21:00.265 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:00 +0000 (0:00:00.100) 0:01:33.732 ********* 2020-06-04T08:21:01.084 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-06-04T08:21:01.090 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"cache_update_time": 1591258833, "cache_updated": false, "changed": false} 2020-06-04T08:21:01.141 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:01 +0000 (0:00:00.875) 0:01:34.608 ********* 2020-06-04T08:21:01.184 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-06-04T08:21:01.222 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:01 +0000 (0:00:00.081) 0:01:34.689 ********* 2020-06-04T08:21:01.224 INFO:teuthology.task.ansible.out: 2020-06-04T08:21:01.265 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-06-04T08:21:01.267 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1591258834, "cache_updated": false, "changed": false} 2020-06-04T08:21:01.316 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:01 +0000 (0:00:00.094) 0:01:34.783 ********* 2020-06-04T08:21:01.371 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-06-04T08:21:01.410 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:01 +0000 (0:00:00.093) 0:01:34.877 ********* 2020-06-04T08:21:01.445 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-06-04T08:21:01.458 INFO:teuthology.task.ansible.out: [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi196.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.035171", "end": "2020-06-04 08:21:01.353056", "rc": 0, "start": "2020-06-04 08:21:01.317885", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-04T08:21:01.496 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:01 +0000 (0:00:00.083) 0:01:34.961 ********* 2020-06-04T08:21:01.651 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.036555", "end": "2020-06-04 08:21:02.022816", "rc": 0, "start": "2020-06-04 08:21:01.986261", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-04T08:21:01.702 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:01 +0000 (0:00:00.208) 0:01:35.169 ********* 2020-06-04T08:21:04.307 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-06-04T08:21:04.309 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"cache_update_time": 1591258833, "cache_updated": false, "changed": false} 2020-06-04T08:21:04.372 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:04 +0000 (0:00:02.669) 0:01:37.839 ********* 2020-06-04T08:21:04.854 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1591258834, "cache_updated": false, "changed": false} 2020-06-04T08:21:04.892 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:04 +0000 (0:00:00.520) 0:01:38.360 ********* 2020-06-04T08:21:04.893 INFO:teuthology.task.ansible.out: 2020-06-04T08:21:13.983 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-06-04T08:21:13.992 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 50 not upgraded.\nAfter this operation, 47.5 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137515 files and directorie 2020-06-04T08:21:13.994 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.23) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.23) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.23) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.23) ...\r\nRemoving librbd1 (12.2.12-0ubuntu0.18.04.5) ...\r\nRemoving librados2 (12.2.12-0ubuntu0.18.04.5) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 50 not upgraded.", "After this operation, 47.5 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2020-06-04T08:21:13.996 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137515 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.23) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.23) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.23) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.23) ...", "Removing librbd1 (12.2.12-0ubuntu0.18.04.5) ...", "Removing librados2 (12.2.12-0ubuntu0.18.04.5) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."]} 2020-06-04T08:21:14.037 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:14 +0000 (0:00:09.142) 0:01:47.502 ********* 2020-06-04T08:21:14.223 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 50 not upgraded.\nAfter this operation, 47.5 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137515 files and directorie 2020-06-04T08:21:14.226 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.23) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.23) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.23) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.23) ...\r\nRemoving librbd1 (12.2.12-0ubuntu0.18.04.5) ...\r\nRemoving librados2 (12.2.12-0ubuntu0.18.04.5) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 50 not upgraded.", "After this operation, 47.5 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2020-06-04T08:21:14.228 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137515 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.23) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.23) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.23) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.23) ...", "Removing librbd1 (12.2.12-0ubuntu0.18.04.5) ...", "Removing librados2 (12.2.12-0ubuntu0.18.04.5) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."]} 2020-06-04T08:21:14.289 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:14 +0000 (0:00:00.254) 0:01:47.757 ********* 2020-06-04T08:21:22.843 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-06-04T08:21:22.847 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1591258834, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following additional packages will be installed:\n apt-utils libapt-pkg5.0\nSuggested packages:\n apt-doc aptitude | synaptic | wajig\nThe following packages will be upgraded:\n apt apt-utils libapt-pkg5.0\n3 upgraded, 0 newly installed, 0 to remove and 47 not upgraded.\nNeed to get 2214 kB of archives.\nAfter this operation, 1024 B of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-pkg5.0 amd64 1.6.12ubuntu0.1 [807 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt amd64 1.6.12ubuntu0.1 [1201 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.12ubuntu0.1 [207 kB]\nFetched 2214 kB in 1s (2596 kB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database . 2020-06-04T08:21:22.849 INFO:teuthology.task.ansible.out:.. 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137373 files and directories currently installed.)\r\nPreparing to unpack .../libapt-pkg5.0_1.6.12ubuntu0.1_amd64.deb ...\r\nUnpacking libapt-pkg5.0:amd64 (1.6.12ubuntu0.1) over (1.6.12) ...\r\nSetting up libapt-pkg5.0:amd64 (1.6.12ubuntu0.1) ...\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137373 files and directories currently installed.)\r\nPreparing to unpack .../apt_1.6.12ubuntu0.1_amd64.deb ...\r\nUnpacking apt (1.6.12ubuntu0.1) over (1.6.12) ...\r\nSetting up apt (1.6.12ubuntu0.1) ...\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading da 2020-06-04T08:21:22.851 INFO:teuthology.task.ansible.out:tabase ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137373 files and directories currently installed.)\r\nPreparing to unpack .../apt-utils_1.6.12ubuntu0.1_amd64.deb ...\r\nUnpacking apt-utils (1.6.12ubuntu0.1) over (1.6.12) ...\r\nSetting up apt-utils (1.6.12ubuntu0.1) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following additional packages will be installed:", " apt-utils libapt-pkg5.0", "Suggested packages:", " apt-doc aptitude | synaptic | wajig", "The following packages will be upgraded:", " apt apt-utils libapt-pkg5.0", "3 upgraded, 0 newly installed, 0 to remove and 47 not upgraded.", "Need to get 2214 kB of archives.", "After this operation, 1024 B of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-pkg5.0 amd64 1.6.12ubuntu0.1 [807 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt amd64 1.6.12ubuntu0.1 [1 2020-06-04T08:21:22.853 INFO:teuthology.task.ansible.out:201 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.12ubuntu0.1 [207 kB]", "Fetched 2214 kB in 1s (2596 kB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137373 files and directories currently installed.)", "Preparing to unpack .../libapt-pkg5.0_1.6.12ubuntu0.1_amd64.deb ...", "Unpacking libapt-pkg5.0:amd64 (1.6.12ubuntu0.1) over (1.6.12) ...", "Setting up libapt-pkg5.0:amd64 (1.6.12ubuntu0.1) ...", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137373 files and directories currently installed.)", "Preparing to unpack .../apt_1.6.12ubuntu0.1_amd64.deb ...", "Unpacking apt (1.6.12ubuntu0.1) over (1.6.12) ...", "Setting up apt (1.6.12ubuntu0.1) ...", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database 2020-06-04T08:21:22.855 INFO:teuthology.task.ansible.out: ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137373 files and directories currently installed.)", "Preparing to unpack .../apt-utils_1.6.12ubuntu0.1_amd64.deb ...", "Unpacking apt-utils (1.6.12ubuntu0.1) over (1.6.12) ...", "Setting up apt-utils (1.6.12ubuntu0.1) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."]} 2020-06-04T08:21:22.901 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:22 +0000 (0:00:08.611) 0:01:56.368 ********* 2020-06-04T08:21:22.954 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-06-04T08:21:22.956 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:23.030 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:23 +0000 (0:00:00.128) 0:01:56.497 ********* 2020-06-04T08:21:24.092 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-06-04T08:21:24.095 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => {"cache_update_time": 1591258833, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following additional packages will be installed:\n apt-utils libapt-pkg5.0\nSuggested packages:\n apt-doc aptitude | synaptic | wajig\nThe following packages will be upgraded:\n apt apt-utils libapt-pkg5.0\n3 upgraded, 0 newly installed, 0 to remove and 47 not upgraded.\nNeed to get 2214 kB of archives.\nAfter this operation, 1024 B of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-pkg5.0 amd64 1.6.12ubuntu0.1 [807 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt amd64 1.6.12ubuntu0.1 [1201 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.12ubuntu0.1 [207 kB]\nFetched 2214 kB in 1s (2536 kB/s)\n(Reading database ... \r(Reading database ... 5%\r(Reading database . 2020-06-04T08:21:24.098 INFO:teuthology.task.ansible.out:.. 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137373 files and directories currently installed.)\r\nPreparing to unpack .../libapt-pkg5.0_1.6.12ubuntu0.1_amd64.deb ...\r\nUnpacking libapt-pkg5.0:amd64 (1.6.12ubuntu0.1) over (1.6.12) ...\r\nSetting up libapt-pkg5.0:amd64 (1.6.12ubuntu0.1) ...\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137373 files and directories currently installed.)\r\nPreparing to unpack .../apt_1.6.12ubuntu0.1_amd64.deb ...\r\nUnpacking apt (1.6.12ubuntu0.1) over (1.6.12) ...\r\nSetting up apt (1.6.12ubuntu0.1) ...\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading da 2020-06-04T08:21:24.100 INFO:teuthology.task.ansible.out:tabase ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137373 files and directories currently installed.)\r\nPreparing to unpack .../apt-utils_1.6.12ubuntu0.1_amd64.deb ...\r\nUnpacking apt-utils (1.6.12ubuntu0.1) over (1.6.12) ...\r\nSetting up apt-utils (1.6.12ubuntu0.1) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following additional packages will be installed:", " apt-utils libapt-pkg5.0", "Suggested packages:", " apt-doc aptitude | synaptic | wajig", "The following packages will be upgraded:", " apt apt-utils libapt-pkg5.0", "3 upgraded, 0 newly installed, 0 to remove and 47 not upgraded.", "Need to get 2214 kB of archives.", "After this operation, 1024 B of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libapt-pkg5.0 amd64 1.6.12ubuntu0.1 [807 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt amd64 1.6.12ubuntu0.1 [1 2020-06-04T08:21:24.102 INFO:teuthology.task.ansible.out:201 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 apt-utils amd64 1.6.12ubuntu0.1 [207 kB]", "Fetched 2214 kB in 1s (2536 kB/s)", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137373 files and directories currently installed.)", "Preparing to unpack .../libapt-pkg5.0_1.6.12ubuntu0.1_amd64.deb ...", "Unpacking libapt-pkg5.0:amd64 (1.6.12ubuntu0.1) over (1.6.12) ...", "Setting up libapt-pkg5.0:amd64 (1.6.12ubuntu0.1) ...", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137373 files and directories currently installed.)", "Preparing to unpack .../apt_1.6.12ubuntu0.1_amd64.deb ...", "Unpacking apt (1.6.12ubuntu0.1) over (1.6.12) ...", "Setting up apt (1.6.12ubuntu0.1) ...", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database 2020-06-04T08:21:24.111 INFO:teuthology.task.ansible.out: ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137373 files and directories currently installed.)", "Preparing to unpack .../apt-utils_1.6.12ubuntu0.1_amd64.deb ...", "Unpacking apt-utils (1.6.12ubuntu0.1) over (1.6.12) ...", "Setting up apt-utils (1.6.12ubuntu0.1) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."]} 2020-06-04T08:21:24.134 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:24 +0000 (0:00:01.104) 0:01:57.602 ********* 2020-06-04T08:21:24.174 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-06-04T08:21:24.176 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:24.224 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:24 +0000 (0:00:00.090) 0:01:57.692 ********* 2020-06-04T08:21:32.023 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-06-04T08:21:32.027 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1591258834, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 47 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.5 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.5 [2704 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.5 [922 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.26 [39.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.26 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.26 [5178 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.26 [870 kB]\nFetched 10.4 MB in 1s (9683 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2020-06-04T08:21:32.029 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137373 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.5_amd64.deb ...\r\nUnpacking librados2 (12.2.12-0ubuntu0.18.04.5) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.5_amd64.deb ...\r\nUnpacking librbd1 (12.2.12-0ubuntu0.18.04.5) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.26) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.26) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.26) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.26) ...\r\nSetting up librados2 (12.2.12-0ubuntu0.18.04.5) ...\r\nSetting up librbd1 (12.2.12-0ubuntu0.18.04.5) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.26) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.26) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.26) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.26) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2020-06-04T08:21:32.031 INFO:teuthology.task.ansible.out:u1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 47 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.5 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.5 [2704 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.5 [922 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.26 [39.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.26 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.26 [5178 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.26 [870 kB]", "Fetched 10.4 MB in 1s (9683 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Rea 2020-06-04T08:21:32.035 INFO:teuthology.task.ansible.out:ding database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137373 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.5_amd64.deb ...", "Unpacking librados2 (12.2.12-0ubuntu0.18.04.5) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.5_amd64.deb ...", "Unpacking librbd1 (12.2.12-0ubuntu0.18.04.5) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.26) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.26) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.26) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.26) ...", "Setting up librados2 (12.2.12-0ubuntu0.18.04.5) ...", "Setting up librbd1 (12.2.12-0ubuntu0.18.04.5) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.26) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.26) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.26) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.26) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."]} 2020-06-04T08:21:32.085 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:32 +0000 (0:00:07.856) 0:02:05.548 ********* 2020-06-04T08:21:32.982 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1591258834, "cache_updated": false, "changed": false} 2020-06-04T08:21:33.019 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:33 +0000 (0:00:00.933) 0:02:06.482 ********* 2020-06-04T08:21:33.175 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-06-04T08:21:33.176 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-06-04T08:21:33.223 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:33 +0000 (0:00:00.208) 0:02:06.691 ********* 2020-06-04T08:21:33.254 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-06-04T08:21:33.256 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:33.293 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:33 +0000 (0:00:00.069) 0:02:06.760 ********* 2020-06-04T08:21:33.385 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-06-04T08:21:33.387 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => {"cache_update_time": 1591258833, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 47 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.5 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.5 [2704 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.5 [922 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.26 [39.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.26 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.26 [5178 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.26 [870 kB]\nFetched 10.4 MB in 1s (9267 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database .. 2020-06-04T08:21:33.389 INFO:teuthology.task.ansible.out:. 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137373 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.5_amd64.deb ...\r\nUnpacking librados2 (12.2.12-0ubuntu0.18.04.5) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.5_amd64.deb ...\r\nUnpacking librbd1 (12.2.12-0ubuntu0.18.04.5) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.26) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.26) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.26) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.26) ...\r\nSetting up librados2 (12.2.12-0ubuntu0.18.04.5) ...\r\nSetting up librbd1 (12.2.12-0ubuntu0.18.04.5) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.26) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.26) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.26) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.26) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubunt 2020-06-04T08:21:33.390 INFO:teuthology.task.ansible.out:u1) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 47 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.5 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.12-0ubuntu0.18.04.5 [2704 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.12-0ubuntu0.18.04.5 [922 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.26 [39.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.26 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.26 [5178 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.26 [870 kB]", "Fetched 10.4 MB in 1s (9267 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Rea 2020-06-04T08:21:33.392 INFO:teuthology.task.ansible.out:ding database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137373 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.12-0ubuntu0.18.04.5_amd64.deb ...", "Unpacking librados2 (12.2.12-0ubuntu0.18.04.5) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.12-0ubuntu0.18.04.5_amd64.deb ...", "Unpacking librbd1 (12.2.12-0ubuntu0.18.04.5) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.26) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.26) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.26) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.26_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.26) ...", "Setting up librados2 (12.2.12-0ubuntu0.18.04.5) ...", "Setting up librbd1 (12.2.12-0ubuntu0.18.04.5) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.26) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.26) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.26) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.26) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1) ..."]} 2020-06-04T08:21:33.438 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:33 +0000 (0:00:00.145) 0:02:06.906 ********* 2020-06-04T08:21:33.468 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-06-04T08:21:33.509 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:33 +0000 (0:00:00.070) 0:02:06.976 ********* 2020-06-04T08:21:33.882 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-06-04T08:21:33.884 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-06-04T08:21:33.928 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:33 +0000 (0:00:00.418) 0:02:07.395 ********* 2020-06-04T08:21:34.119 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-06-04T08:21:34.121 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1215, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-06-04T08:21:34.164 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:34 +0000 (0:00:00.236) 0:02:07.631 ********* 2020-06-04T08:21:34.292 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-06-04T08:21:34.294 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"cache_update_time": 1591258833, "cache_updated": false, "changed": false} 2020-06-04T08:21:34.342 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:34 +0000 (0:00:00.178) 0:02:07.809 ********* 2020-06-04T08:21:34.351 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** ok: [smithi060.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-04T08:21:34.404 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:34 +0000 (0:00:00.060) 0:02:07.870 ********* 2020-06-04T08:21:34.503 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-06-04T08:21:34.504 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2020-06-04T08:21:34.542 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:34 +0000 (0:00:00.139) 0:02:08.010 ********* 2020-06-04T08:21:34.584 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-06-04T08:21:34.586 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:34.620 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:34 +0000 (0:00:00.077) 0:02:08.088 ********* 2020-06-04T08:21:34.728 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-06-04T08:21:34.730 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2020-06-04 08:18:50 UTC", "ActiveEnterTimestampMonotonic": "20042252", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target basic.target -.mount system.slice systemd-journald.socket sysinit.target systemd-tmpfiles-setup.service remote-fs.target nss-lookup.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-06-04 08:18:50 UTC", "AssertTimestampMonotonic": "19459059", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-06-04 08:18:50 UTC", "ConditionTimestampMonotonic": "19459059", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-06-04T08:21:34.732 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1338", "ExecMainStartTimestamp": "Thu 2020-06-04 08:18:50 UTC", "ExecMainStartTimestampMonotonic": "20042241", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-06-04 08:18:50 UTC", "InactiveExitTimestampMonotonic": "19459760", "InvocationID": "b2844fc0c6f84ccaa9667ec42fd1d4f5", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-06-04T08:21:34.735 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1338", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-06-04T08:21:34.736 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-06-04 08:18:50 UTC", "StateChangeTimestampMonotonic": "20042252", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-06-04 08:18:50 UTC", "WatchdogTimestampMonotonic": "20042251", "WatchdogUSec": "0"}} 2020-06-04T08:21:34.777 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:34 +0000 (0:00:00.156) 0:02:08.244 ********* 2020-06-04T08:21:34.787 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-06-04T08:21:34.789 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2020-06-04T08:21:34.843 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:34 +0000 (0:00:00.066) 0:02:08.311 ********* 2020-06-04T08:21:34.863 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:34.909 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:34 +0000 (0:00:00.065) 0:02:08.376 ********* 2020-06-04T08:21:34.954 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:34.991 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:34 +0000 (0:00:00.082) 0:02:08.458 ********* 2020-06-04T08:21:35.020 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-06-04T08:21:35.022 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:35.068 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:35 +0000 (0:00:00.077) 0:02:08.536 ********* 2020-06-04T08:21:35.110 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-06-04T08:21:35.112 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:35.151 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:35 +0000 (0:00:00.082) 0:02:08.619 ********* 2020-06-04T08:21:35.201 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:35.248 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:35 +0000 (0:00:00.096) 0:02:08.715 ********* 2020-06-04T08:21:35.269 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-06-04T08:21:35.271 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2020-06-04T08:21:35.309 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:35 +0000 (0:00:00.060) 0:02:08.776 ********* 2020-06-04T08:21:35.319 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-06-04T08:21:35.321 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:35.363 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:35 +0000 (0:00:00.053) 0:02:08.830 ********* 2020-06-04T08:21:35.422 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-06-04T08:21:35.425 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:35.473 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:35 +0000 (0:00:00.102) 0:02:08.933 ********* 2020-06-04T08:21:35.498 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-06-04T08:21:35.500 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1215, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-06-04T08:21:35.548 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:35 +0000 (0:00:00.083) 0:02:09.016 ********* 2020-06-04T08:21:35.558 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-06-04T08:21:35.560 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:35.598 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:35 +0000 (0:00:00.049) 0:02:09.065 ********* 2020-06-04T08:21:35.641 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-04T08:21:35.643 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:35.684 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:35 +0000 (0:00:00.086) 0:02:09.151 ********* 2020-06-04T08:21:35.704 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** ok: [smithi196.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-04T08:21:35.741 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:35 +0000 (0:00:00.056) 0:02:09.208 ********* 2020-06-04T08:21:35.758 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:35.794 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:35 +0000 (0:00:00.052) 0:02:09.261 ********* 2020-06-04T08:21:35.838 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:35.872 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:35 +0000 (0:00:00.078) 0:02:09.340 ********* 2020-06-04T08:21:35.906 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-06-04T08:21:35.909 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:35.954 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:35 +0000 (0:00:00.075) 0:02:09.416 ********* 2020-06-04T08:21:35.979 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-06-04T08:21:35.981 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:36.038 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:36 +0000 (0:00:00.090) 0:02:09.506 ********* 2020-06-04T08:21:36.072 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-06-04T08:21:36.074 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Thu 2020-06-04 08:19:10 UTC", "ActiveEnterTimestampMonotonic": "18839409", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "remote-fs.target systemd-journald.socket network.target system.slice systemd-tmpfiles-setup.service basic.target -.mount sysinit.target nss-lookup.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-06-04 08:19:09 UTC", "AssertTimestampMonotonic": "18240311", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-06-04 08:19:09 UTC", "ConditionTimestampMonotonic": "18240311", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2020-06-04T08:21:36.076 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1263", "ExecMainStartTimestamp": "Thu 2020-06-04 08:19:10 UTC", "ExecMainStartTimestampMonotonic": "18839398", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-06-04 08:19:09 UTC", "InactiveExitTimestampMonotonic": "18241738", "InvocationID": "f96bd642f7c84212a6670c8e86768bb7", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2020-06-04T08:21:36.078 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1263", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2020-06-04T08:21:36.080 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-06-04 08:19:10 UTC", "StateChangeTimestampMonotonic": "18839409", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-06-04 08:19:10 UTC", "WatchdogTimestampMonotonic": "18839408", "WatchdogUSec": "0"}} 2020-06-04T08:21:36.112 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:36 +0000 (0:00:00.073) 0:02:09.579 ********* 2020-06-04T08:21:36.121 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-06-04T08:21:36.123 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:36.161 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:36 +0000 (0:00:00.049) 0:02:09.629 ********* 2020-06-04T08:21:36.172 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-06-04T08:21:36.176 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:36.213 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:36 +0000 (0:00:00.051) 0:02:09.680 ********* 2020-06-04T08:21:36.224 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-06-04T08:21:36.226 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:36.269 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:36 +0000 (0:00:00.055) 0:02:09.736 ********* 2020-06-04T08:21:36.279 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-06-04T08:21:36.280 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:36.338 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:36 +0000 (0:00:00.069) 0:02:09.805 ********* 2020-06-04T08:21:36.348 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-06-04T08:21:36.350 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:36.391 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:36 +0000 (0:00:00.052) 0:02:09.858 ********* 2020-06-04T08:21:36.403 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-06-04T08:21:36.405 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:36.444 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:36 +0000 (0:00:00.053) 0:02:09.911 ********* 2020-06-04T08:21:36.453 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-06-04T08:21:36.455 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:36.501 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:36 +0000 (0:00:00.057) 0:02:09.969 ********* 2020-06-04T08:21:36.510 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-06-04T08:21:36.512 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:36.552 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:36 +0000 (0:00:00.051) 0:02:10.020 ********* 2020-06-04T08:21:36.561 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-06-04T08:21:36.562 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:36.602 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:36 +0000 (0:00:00.049) 0:02:10.069 ********* 2020-06-04T08:21:36.610 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-06-04T08:21:36.613 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:36.662 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:36 +0000 (0:00:00.059) 0:02:10.129 ********* 2020-06-04T08:21:36.669 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-06-04T08:21:36.671 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:36.713 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:36 +0000 (0:00:00.050) 0:02:10.179 ********* 2020-06-04T08:21:36.720 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-06-04T08:21:36.722 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:36.758 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:36 +0000 (0:00:00.046) 0:02:10.225 ********* 2020-06-04T08:21:36.767 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-06-04T08:21:36.770 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:36.810 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:36 +0000 (0:00:00.047) 0:02:10.273 ********* 2020-06-04T08:21:36.814 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:36.854 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:36 +0000 (0:00:00.048) 0:02:10.321 ********* 2020-06-04T08:21:36.863 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-06-04T08:21:36.865 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:36.911 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:36 +0000 (0:00:00.057) 0:02:10.379 ********* 2020-06-04T08:21:36.922 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-06-04T08:21:36.924 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:36.977 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:36 +0000 (0:00:00.065) 0:02:10.444 ********* 2020-06-04T08:21:36.987 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-06-04T08:21:36.989 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:37.033 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:37 +0000 (0:00:00.055) 0:02:10.500 ********* 2020-06-04T08:21:37.044 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-04T08:21:37.046 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:37.085 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:37 +0000 (0:00:00.051) 0:02:10.552 ********* 2020-06-04T08:21:37.094 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-06-04T08:21:37.097 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:37.156 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:37 +0000 (0:00:00.069) 0:02:10.621 ********* 2020-06-04T08:21:37.165 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-06-04T08:21:37.167 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:37.204 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:37 +0000 (0:00:00.049) 0:02:10.671 ********* 2020-06-04T08:21:37.214 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-06-04T08:21:37.216 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:37.258 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:37 +0000 (0:00:00.050) 0:02:10.722 ********* 2020-06-04T08:21:37.266 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:37.309 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:37 +0000 (0:00:00.054) 0:02:10.776 ********* 2020-06-04T08:21:37.320 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:37.362 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:37 +0000 (0:00:00.049) 0:02:10.825 ********* 2020-06-04T08:21:37.370 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-06-04T08:21:37.372 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:37.422 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:37 +0000 (0:00:00.062) 0:02:10.888 ********* 2020-06-04T08:21:37.431 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-06-04T08:21:37.433 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:37.483 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:37 +0000 (0:00:00.061) 0:02:10.950 ********* 2020-06-04T08:21:37.495 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:37.547 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:37 +0000 (0:00:00.064) 0:02:11.014 ********* 2020-06-04T08:21:37.615 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:37.667 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:37 +0000 (0:00:00.119) 0:02:11.134 ********* 2020-06-04T08:21:37.675 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-06-04T08:21:37.677 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-06-04T08:21:37.742 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:37 +0000 (0:00:00.075) 0:02:11.210 ********* 2020-06-04T08:21:37.754 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-06-04T08:21:37.756 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:37.797 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:37 +0000 (0:00:00.054) 0:02:11.264 ********* 2020-06-04T08:21:37.840 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-06-04T08:21:37.842 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:37.883 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:37 +0000 (0:00:00.085) 0:02:11.350 ********* 2020-06-04T08:21:37.926 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:37.962 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:37 +0000 (0:00:00.079) 0:02:11.429 ********* 2020-06-04T08:21:38.003 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-06-04T08:21:38.009 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:38.040 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:38 +0000 (0:00:00.078) 0:02:11.507 ********* 2020-06-04T08:21:38.091 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-06-04T08:21:38.093 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:38.139 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:38 +0000 (0:00:00.098) 0:02:11.607 ********* 2020-06-04T08:21:38.148 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-06-04T08:21:38.149 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-06-04T08:21:38.188 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:38 +0000 (0:00:00.048) 0:02:11.655 ********* 2020-06-04T08:21:38.197 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-06-04T08:21:38.198 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:38.254 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:38 +0000 (0:00:00.066) 0:02:11.721 ********* 2020-06-04T08:21:38.263 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-06-04T08:21:38.265 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:38.311 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:38 +0000 (0:00:00.056) 0:02:11.778 ********* 2020-06-04T08:21:38.319 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-06-04T08:21:38.321 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:38.365 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:38 +0000 (0:00:00.054) 0:02:11.833 ********* 2020-06-04T08:21:38.417 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-06-04T08:21:38.424 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:38.475 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:38 +0000 (0:00:00.109) 0:02:11.942 ********* 2020-06-04T08:21:38.515 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-06-04T08:21:38.517 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:38.569 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:38 +0000 (0:00:00.093) 0:02:12.036 ********* 2020-06-04T08:21:38.608 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:38.661 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:38 +0000 (0:00:00.092) 0:02:12.128 ********* 2020-06-04T08:21:38.704 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-06-04T08:21:38.706 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:38.741 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:38 +0000 (0:00:00.079) 0:02:12.208 ********* 2020-06-04T08:21:38.755 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-06-04T08:21:38.757 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-06-04T08:21:38.798 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:38 +0000 (0:00:00.056) 0:02:12.264 ********* 2020-06-04T08:21:38.814 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:38.851 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:38 +0000 (0:00:00.054) 0:02:12.319 ********* 2020-06-04T08:21:39.021 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-06-04T08:21:39.023 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-06-04T08:21:39.059 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:39 +0000 (0:00:00.207) 0:02:12.526 ********* 2020-06-04T08:21:39.215 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-06-04T08:21:39.279 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:39 +0000 (0:00:00.220) 0:02:12.747 ********* 2020-06-04T08:21:39.456 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-06-04T08:21:39.496 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:39 +0000 (0:00:00.216) 0:02:12.964 ********* 2020-06-04T08:21:39.540 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-06-04T08:21:39.542 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:39.596 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:39 +0000 (0:00:00.100) 0:02:13.064 ********* 2020-06-04T08:21:39.667 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-06-04T08:21:39.669 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-06-04T08:21:39.712 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:39 +0000 (0:00:00.115) 0:02:13.179 ********* 2020-06-04T08:21:39.879 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-06-04T08:21:39.881 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-04T08:21:39.937 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:39 +0000 (0:00:00.225) 0:02:13.404 ********* 2020-06-04T08:21:39.968 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-06-04T08:21:39.970 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-06-04T08:21:39.972 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:40.007 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:40 +0000 (0:00:00.070) 0:02:13.475 ********* 2020-06-04T08:21:40.046 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:40 +0000 (0:00:00.038) 0:02:13.513 ********* 2020-06-04T08:21:40.054 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-06-04T08:21:40.056 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:40.095 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:40 +0000 (0:00:00.049) 0:02:13.562 ********* 2020-06-04T08:21:40.125 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:40.164 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:40 +0000 (0:00:00.069) 0:02:13.632 ********* 2020-06-04T08:21:40.193 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-06-04T08:21:40.195 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:40.247 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:40 +0000 (0:00:00.081) 0:02:13.713 ********* 2020-06-04T08:21:40.289 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:40.325 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:40 +0000 (0:00:00.079) 0:02:13.793 ********* 2020-06-04T08:21:40.376 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-04T08:21:40.378 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-06-04T08:21:40.426 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:40 +0000 (0:00:00.099) 0:02:13.892 ********* 2020-06-04T08:21:40.436 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-06-04T08:21:40.438 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-06-04T08:21:40.474 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:40 +0000 (0:00:00.048) 0:02:13.941 ********* 2020-06-04T08:21:40.495 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-06-04T08:21:40.497 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-06-04T08:21:40.508 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-06-04T08:21:40.510 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-06-04T08:21:40.522 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-06-04T08:21:40.536 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-06-04T08:21:40.541 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-06-04T08:21:40.588 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-06-04T08:21:40.590 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-06-04T08:21:40.591 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-06-04T08:21:40.591 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-06-04T08:21:40.602 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-06-04T08:21:40.691 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:40 +0000 (0:00:00.216) 0:02:14.158 ********* 2020-06-04T08:21:40.883 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-06-04T08:21:40.890 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-06-04T08:21:40.979 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:40 +0000 (0:00:00.288) 0:02:14.446 ********* 2020-06-04T08:21:40.990 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-06-04T08:21:40.992 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004547", "end": "2020-06-04 08:21:40.957268", "rc": 0, "start": "2020-06-04 08:21:40.952721", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-06-04T08:21:41.062 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:41 +0000 (0:00:00.083) 0:02:14.530 ********* 2020-06-04T08:21:41.111 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:41.156 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:41 +0000 (0:00:00.094) 0:02:14.624 ********* 2020-06-04T08:21:41.167 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-06-04T08:21:41.169 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-04T08:21:41.222 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:41 +0000 (0:00:00.064) 0:02:14.689 ********* 2020-06-04T08:21:41.230 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-06-04T08:21:41.232 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:41.271 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:41 +0000 (0:00:00.049) 0:02:14.739 ********* 2020-06-04T08:21:41.280 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-06-04T08:21:41.282 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:41.319 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:41 +0000 (0:00:00.048) 0:02:14.787 ********* 2020-06-04T08:21:41.329 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-06-04T08:21:41.332 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:41.368 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:41 +0000 (0:00:00.048) 0:02:14.836 ********* 2020-06-04T08:21:41.377 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-06-04T08:21:41.378 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:41.432 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:41 +0000 (0:00:00.063) 0:02:14.899 ********* 2020-06-04T08:21:41.473 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-06-04T08:21:41.474 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:41.520 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:41 +0000 (0:00:00.088) 0:02:14.988 ********* 2020-06-04T08:21:41.574 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:41.610 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:41 +0000 (0:00:00.089) 0:02:15.077 ********* 2020-06-04T08:21:41.658 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-06-04T08:21:41.660 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:41.696 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:41 +0000 (0:00:00.086) 0:02:15.164 ********* 2020-06-04T08:21:41.736 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-04T08:21:41.738 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-06-04T08:21:41.785 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:41 +0000 (0:00:00.087) 0:02:15.251 ********* 2020-06-04T08:21:41.862 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi196.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2020-06-04T08:21:41.874 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-06-04T08:21:41.893 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-06-04T08:21:41.906 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-06-04T08:21:41.916 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} ok: [smithi196.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-06-04T08:21:41.937 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-06-04T08:21:41.939 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-06-04T08:21:41.948 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-06-04T08:21:41.959 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-06-04T08:21:41.970 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-06-04T08:21:41.981 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-06-04T08:21:41.983 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-06-04T08:21:41.993 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-06-04T08:21:42.075 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:42 +0000 (0:00:00.290) 0:02:15.542 ********* 2020-06-04T08:21:42.302 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1591258834, "cache_updated": false, "changed": false} 2020-06-04T08:21:42.343 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:42 +0000 (0:00:00.267) 0:02:15.810 ********* 2020-06-04T08:21:42.363 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi196.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004597", "end": "2020-06-04 08:21:42.304612", "rc": 0, "start": "2020-06-04 08:21:42.300015", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-06-04T08:21:42.398 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:42 +0000 (0:00:00.054) 0:02:15.865 ********* 2020-06-04T08:21:42.407 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-06-04T08:21:42.409 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:42.456 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:42 +0000 (0:00:00.058) 0:02:15.923 ********* 2020-06-04T08:21:42.466 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-06-04T08:21:42.468 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:42.512 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:42 +0000 (0:00:00.053) 0:02:15.977 ********* 2020-06-04T08:21:42.530 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-06-04T08:21:42.532 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'block_used': 13741041, 'uuid': '81cc74cc-5dd2-4c99-9728-93a5b2dec7d8', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226354067, 'size_available': 927146258432, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149294, 'block_size': 4096, 'inode_available': 60905682}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226354067, "block_size": 4096, "block_total": 240095108, "block_used": 13741041, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905682, "inode_total": 61054976, "inode_used": 149294, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927146258432, "size_total": 983429562368, "uuid": "81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"}} 2020-06-04T08:21:42.569 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:42 +0000 (0:00:00.059) 0:02:16.037 ********* 2020-06-04T08:21:42.579 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-06-04T08:21:42.581 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:42.619 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:42 +0000 (0:00:00.049) 0:02:16.087 ********* 2020-06-04T08:21:42.681 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-06-04T08:21:42.684 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:42.720 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:42 +0000 (0:00:00.099) 0:02:16.186 ********* 2020-06-04T08:21:42.740 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-06-04T08:21:42.744 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.008010", "end": "2020-06-04 08:21:43.084955", "rc": 0, "start": "2020-06-04 08:21:43.076945", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-04T08:21:42.783 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:42 +0000 (0:00:00.064) 0:02:16.251 ********* 2020-06-04T08:21:42.838 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-06-04T08:21:42.840 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'block_used': 13741041, 'uuid': '81cc74cc-5dd2-4c99-9728-93a5b2dec7d8', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226354067, 'size_available': 927146258432, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149294, 'block_size': 4096, 'inode_available': 60905682}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226354067, "block_size": 4096, "block_total": 240095108, "block_used": 13741041, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905682, "inode_total": 61054976, "inode_used": 149294, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927146258432, "size_total": 983429562368, "uuid": "81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:42.901 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:42 +0000 (0:00:00.117) 0:02:16.369 ********* 2020-06-04T08:21:42.952 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-06-04T08:21:42.992 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:42 +0000 (0:00:00.090) 0:02:16.459 ********* 2020-06-04T08:21:43.055 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-06-04T08:21:43.056 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:43.648 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-06-04T08:21:43.654 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"cache_update_time": 1591258833, "cache_updated": false, "changed": false} 2020-06-04T08:21:43.702 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:43 +0000 (0:00:00.709) 0:02:17.169 ********* 2020-06-04T08:21:43.754 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-06-04T08:21:43.756 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:43.792 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:43 +0000 (0:00:00.089) 0:02:17.259 ********* 2020-06-04T08:21:43.841 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-06-04T08:21:43.843 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item={'block_used': 13741035, 'uuid': '81cc74cc-5dd2-4c99-9728-93a5b2dec7d8', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226354073, 'size_available': 927146283008, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149293, 'block_size': 4096, 'inode_available': 60905683}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226354073, "block_size": 4096, "block_total": 240095108, "block_used": 13741035, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905683, "inode_total": 61054976, "inode_used": 149293, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927146283008, "size_total": 983429562368, "uuid": "81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"}} 2020-06-04T08:21:43.894 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:43 +0000 (0:00:00.101) 0:02:17.361 ********* 2020-06-04T08:21:44.060 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-06-04T08:21:44.062 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.010008", "end": "2020-06-04 08:21:44.003136", "rc": 0, "start": "2020-06-04 08:21:43.993128", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-04T08:21:44.103 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:44 +0000 (0:00:00.209) 0:02:17.570 ********* 2020-06-04T08:21:44.152 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-06-04T08:21:44.154 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'block_used': 13741035, 'uuid': '81cc74cc-5dd2-4c99-9728-93a5b2dec7d8', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226354073, 'size_available': 927146283008, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149293, 'block_size': 4096, 'inode_available': 60905683}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226354073, "block_size": 4096, "block_total": 240095108, "block_used": 13741035, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905683, "inode_total": 61054976, "inode_used": 149293, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927146283008, "size_total": 983429562368, "uuid": "81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:44.196 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:44 +0000 (0:00:00.093) 0:02:17.664 ********* 2020-06-04T08:21:44.225 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-06-04T08:21:44.261 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:44 +0000 (0:00:00.064) 0:02:17.728 ********* 2020-06-04T08:21:44.326 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** skipping: [smithi196.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:44.760 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-yBiFOqFg9SKod1vGrhd2d79I6cBeqWrETUKgJitiLGemKGAw4EGVQYKZQmS9Byf2'], 'uuids': ['5f1ea888-e26c-4f9f-b102-e1dbe267f708']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.556117", "end": "2020-06-04 08:21:45.095665", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-yBiFOqFg9SKod1vGrhd2d79I6cBeqWrETUKgJitiLGemKGAw4EGVQYKZQmS9Byf2"], "labels": [], "masters": [], "uuids": ["5f1ea888-e26c-4f9f-b102-e1dbe267f708"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:43.539548", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-04T08:21:44.772 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:44.790 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:46.033 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-yBiFOqFg9SKod1vGrhd2d79I6cBeqWrErG9mXgk6b83kCu15h3P6NVYz6z7zPFEW'], 'uuids': ['eeb75bec-a67a-4b0d-9b51-18bbed0cc7d4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.110115", "end": "2020-06-04 08:21:46.367466", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-yBiFOqFg9SKod1vGrhd2d79I6cBeqWrErG9mXgk6b83kCu15h3P6NVYz6z7zPFEW"], "labels": [], "masters": [], "uuids": ["eeb75bec-a67a-4b0d-9b51-18bbed0cc7d4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:45.257351", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-04T08:21:46.035 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4H4ED', 'wwn-0x5000c500871b1a82'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '81cc74cc-5dd2-4c99-9728-93a5b2dec7d8', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4H4ED-part1', 'wwn-0x5000c500871b1a82-part1'], 'uuids': ['81cc74cc-5dd2-4c99-9728-93a5b2dec7d8']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871b1a82', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4H4ED', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4H4ED", "wwn-0x5000c500871b1a82"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4H4ED-part1", "wwn-0x5000c500871b1a82-part1"], "labels": [], "masters": [], "uuids": ["81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4H4ED", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b1a82"}}, "skip_reason": 2020-06-04T08:21:46.037 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-06-04T08:21:46.112 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-ypiX2xSrkSioOqIB7uUWqfvpyTEo69gpRSu58XrMhRo3DMa7NUpvVivlA9iTGLrS'], 'uuids': ['eac3d426-0ec1-42fd-a1ec-7e9fbe89193f']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.630904", "end": "2020-06-04 08:21:46.059062", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-ypiX2xSrkSioOqIB7uUWqfvpyTEo69gpRSu58XrMhRo3DMa7NUpvVivlA9iTGLrS"], "labels": [], "masters": [], "uuids": ["eac3d426-0ec1-42fd-a1ec-7e9fbe89193f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:44.428158", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-04T08:21:46.134 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:46.146 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:47.396 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-gS6B00-wlZ1-ffnC-leS2-lDxP-6Y02-JkfOsT', 'nvme-INTEL_SSDPED1D480GA_PHMB7476008A480DGN', 'nvme-nvme.8086-50484d42373437363030384134383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_4', 'vg_nvme-lv_1', 'vg_nvme-lv_3', 'vg_nvme-lv_5', 'vg_nvme-lv_2'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.204859", "end": "2020-06-04 08:21:47.724631", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-gS6B00-wlZ1-ffnC-leS2-lDxP-6Y02-JkfOsT", "nvme-INTEL_SSDPED1D480GA_PHMB7476008A480DGN", "nvme-nvme.8086-50484d42373437363030384134383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:46.519772", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now pa 2020-06-04T08:21:47.398 INFO:teuthology.task.ansible.out:rtition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-04T08:21:47.416 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} changed: [smithi196.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-ypiX2xSrkSioOqIB7uUWqfvpyTEo69gpKcIBwQZldhFNI9ayVFMq4YmE0XVfhM7Q'], 'uuids': ['ad89084a-5c57-4de8-9f68-48d9fe090feb']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.150118", "end": "2020-06-04 08:21:47.377267", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-ypiX2xSrkSioOqIB7uUWqfvpyTEo69gpKcIBwQZldhFNI9ayVFMq4YmE0XVfhM7Q"], "labels": [], "masters": [], "uuids": ["ad89084a-5c57-4de8-9f68-48d9fe090feb"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "s 2020-06-04T08:21:47.417 INFO:teuthology.task.ansible.out:as_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:46.227149", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-04T08:21:47.428 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:47.430 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XGPQ', 'wwn-0x5000c50092928bab'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '81cc74cc-5dd2-4c99-9728-93a5b2dec7d8', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XGPQ-part1', 'wwn-0x5000c50092928bab-part1'], 'uuids': ['81cc74cc-5dd2-4c99-9728-93a5b2dec7d8']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092928bab', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XGPQ', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XGPQ", "wwn-0x5000c50092928bab"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XGPQ-part1", "wwn-0x5000c50092928bab-part1"], "labels": [], "masters": [], "uuids": ["81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XGPQ", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092928bab"}}, "skip_reason": 2020-06-04T08:21:47.433 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-06-04T08:21:48.707 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-yBiFOqFg9SKod1vGrhd2d79I6cBeqWrE2ubFO3TwKhdmTHSTaRlngZKlFuv0gbYN'], 'uuids': ['3a092fa5-c0f1-4e05-b845-340c6fda9ea3']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.150398", "end": "2020-06-04 08:21:49.041470", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-yBiFOqFg9SKod1vGrhd2d79I6cBeqWrE2ubFO3TwKhdmTHSTaRlngZKlFuv0gbYN"], "labels": [], "masters": [], "uuids": ["3a092fa5-c0f1-4e05-b845-340c6fda9ea3"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:47.891072", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-04T08:21:48.719 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:48.741 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:48.815 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-EYXAoB-5E1w-suAF-KqgK-kTRk-r3xZ-govjsB', 'nvme-INTEL_SSDPEDMD400G4_CVFT6246000Z400BGN', 'nvme-nvme.8086-43564654363234363030305a34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_4', 'vg_nvme-lv_1', 'vg_nvme-lv_3', 'vg_nvme-lv_5', 'vg_nvme-lv_2'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.225111", "end": "2020-06-04 08:21:48.756383", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-EYXAoB-5E1w-suAF-KqgK-kTRk-r3xZ-govjsB", "nvme-INTEL_SSDPEDMD400G4_CVFT6246000Z400BGN", "nvme-nvme.8086-43564654363234363030305a34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:47.531272", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2020-06-04T08:21:48.816 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-04T08:21:48.821 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:48.822 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:50.000 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-yBiFOqFg9SKod1vGrhd2d79I6cBeqWrEsD7FenmHjZsXhdD42daEv7N4XuNomOaY'], 'uuids': ['bd2d9608-9cfd-452d-af5e-69a11ce705f9']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.125236", "end": "2020-06-04 08:21:50.334211", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-yBiFOqFg9SKod1vGrhd2d79I6cBeqWrEsD7FenmHjZsXhdD42daEv7N4XuNomOaY"], "labels": [], "masters": [], "uuids": ["bd2d9608-9cfd-452d-af5e-69a11ce705f9"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:49.208975", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-04T08:21:50.011 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:50.120 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-ypiX2xSrkSioOqIB7uUWqfvpyTEo69gpTDR85VraZx8DK1SIn50BBLLWDHM0l4RI'], 'uuids': ['ba1a4c9b-c2aa-4299-8f6c-391001fd94f5']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.158342", "end": "2020-06-04 08:21:50.070611", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-ypiX2xSrkSioOqIB7uUWqfvpyTEo69gpTDR85VraZx8DK1SIn50BBLLWDHM0l4RI"], "labels": [], "masters": [], "uuids": ["ba1a4c9b-c2aa-4299-8f6c-391001fd94f5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:48.912269", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-04T08:21:50.127 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:50.140 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:51.277 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-yBiFOqFg9SKod1vGrhd2d79I6cBeqWrEwBbn3SeFVqX6QptGYeRDMHaXfWCk6iT2'], 'uuids': ['c5f566c4-6058-4ef9-81db-bd73cd9b9965']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.140150", "end": "2020-06-04 08:21:51.616974", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-yBiFOqFg9SKod1vGrhd2d79I6cBeqWrEwBbn3SeFVqX6QptGYeRDMHaXfWCk6iT2"], "labels": [], "masters": [], "uuids": ["c5f566c4-6058-4ef9-81db-bd73cd9b9965"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:50.476824", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-04T08:21:51.348 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:51 +0000 (0:00:07.087) 0:02:24.815 ********* 2020-06-04T08:21:51.402 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-ypiX2xSrkSioOqIB7uUWqfvpyTEo69gpC5ByS5Qtm5Zy4fJUBQ1uMl75GBA02LBE'], 'uuids': ['9baaeb27-ce36-42b8-9327-8b9b6a424350']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.131099", "end": "2020-06-04 08:21:51.353925", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-ypiX2xSrkSioOqIB7uUWqfvpyTEo69gpC5ByS5Qtm5Zy4fJUBQ1uMl75GBA02LBE"], "labels": [], "masters": [], "uuids": ["9baaeb27-ce36-42b8-9327-8b9b6a424350"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:50.222826", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-06-04T08:21:51.429 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} TASK [testnode : Zap all non-root disks] *************************************** skipping: [smithi196.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:51.680 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-06-04T08:21:51.682 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-yBiFOqFg9SKod1vGrhd2d79I6cBeqWrETUKgJitiLGemKGAw4EGVQYKZQmS9Byf2'], 'uuids': ['5f1ea888-e26c-4f9f-b102-e1dbe267f708']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.112634", "end": "2020-06-04 08:21:52.015706", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-yBiFOqFg9SKod1vGrhd2d79I6cBeqWrETUKgJitiLGemKGAw4EGVQYKZQmS9Byf2"], "labels": [], "masters": [], "uuids": ["5f1ea888-e26c-4f9f-b102-e1dbe267f708"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:51.903072", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.099337 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.099337 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2020-06-04T08:21:51.692 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:51.703 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:51.931 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-yBiFOqFg9SKod1vGrhd2d79I6cBeqWrErG9mXgk6b83kCu15h3P6NVYz6z7zPFEW'], 'uuids': ['eeb75bec-a67a-4b0d-9b51-18bbed0cc7d4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.105443", "end": "2020-06-04 08:21:52.275030", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-yBiFOqFg9SKod1vGrhd2d79I6cBeqWrErG9mXgk6b83kCu15h3P6NVYz6z7zPFEW"], "labels": [], "masters": [], "uuids": ["eeb75bec-a67a-4b0d-9b51-18bbed0cc7d4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:52.169587", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.103236 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.103236 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2020-06-04T08:21:51.943 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4H4ED', 'wwn-0x5000c500871b1a82'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '81cc74cc-5dd2-4c99-9728-93a5b2dec7d8', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4H4ED-part1', 'wwn-0x5000c500871b1a82-part1'], 'uuids': ['81cc74cc-5dd2-4c99-9728-93a5b2dec7d8']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871b1a82', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4H4ED', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4H4ED", "wwn-0x5000c500871b1a82"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4H4ED-part1", "wwn-0x5000c500871b1a82-part1"], "labels": [], "masters": [], "uuids": ["81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4H4ED", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b1a82"}}, "skip_reason": 2020-06-04T08:21:51.945 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-06-04T08:21:52.123 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-gS6B00-wlZ1-ffnC-leS2-lDxP-6Y02-JkfOsT', 'nvme-INTEL_SSDPED1D480GA_PHMB7476008A480DGN', 'nvme-nvme.8086-50484d42373437363030384134383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_4', 'vg_nvme-lv_1', 'vg_nvme-lv_3', 'vg_nvme-lv_5', 'vg_nvme-lv_2'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.041740", "end": "2020-06-04 08:21:52.458929", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-gS6B00-wlZ1-ffnC-leS2-lDxP-6Y02-JkfOsT", "nvme-INTEL_SSDPED1D480GA_PHMB7476008A480DGN", "nvme-nvme.8086-50484d42373437363030384134383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:52.417189", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0391003 s, 2.9 GB/s", "stderr_lines": [" 2020-06-04T08:21:52.126 INFO:teuthology.task.ansible.out:110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0391003 s, 2.9 GB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:52.137 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:52.377 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-yBiFOqFg9SKod1vGrhd2d79I6cBeqWrE2ubFO3TwKhdmTHSTaRlngZKlFuv0gbYN'], 'uuids': ['3a092fa5-c0f1-4e05-b845-340c6fda9ea3']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.114969", "end": "2020-06-04 08:21:52.717489", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-yBiFOqFg9SKod1vGrhd2d79I6cBeqWrE2ubFO3TwKhdmTHSTaRlngZKlFuv0gbYN"], "labels": [], "masters": [], "uuids": ["3a092fa5-c0f1-4e05-b845-340c6fda9ea3"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:52.602520", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.112488 s, 1.0 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.112488 s, 1.0 GB/s"], "stdout": "", "stdout_lines": []} 2020-06-04T08:21:52.400 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:52.412 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:52.645 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-yBiFOqFg9SKod1vGrhd2d79I6cBeqWrEsD7FenmHjZsXhdD42daEv7N4XuNomOaY'], 'uuids': ['bd2d9608-9cfd-452d-af5e-69a11ce705f9']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.112108", "end": "2020-06-04 08:21:52.986771", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-yBiFOqFg9SKod1vGrhd2d79I6cBeqWrEsD7FenmHjZsXhdD42daEv7N4XuNomOaY"], "labels": [], "masters": [], "uuids": ["bd2d9608-9cfd-452d-af5e-69a11ce705f9"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:52.874663", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.10752 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.10752 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2020-06-04T08:21:52.657 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:52.702 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi196.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-ypiX2xSrkSioOqIB7uUWqfvpyTEo69gpDrvDblB3T0XJMFr6yJSZs5cfmLZbFQoK'], 'uuids': ['9765944d-02e4-408c-a822-1a32035891e9']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.148310", "end": "2020-06-04 08:21:52.654806", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-ypiX2xSrkSioOqIB7uUWqfvpyTEo69gpDrvDblB3T0XJMFr6yJSZs5cfmLZbFQoK"], "labels": [], "masters": [], "uuids": ["9765944d-02e4-408c-a822-1a32035891e9"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:51.506496", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk u 2020-06-04T08:21:52.704 INFO:teuthology.task.ansible.out:sing fdisk or", "other utilities."]} 2020-06-04T08:21:52.779 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:52 +0000 (0:00:01.431) 0:02:26.247 ********* 2020-06-04T08:21:52.843 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-06-04T08:21:52.845 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:52.890 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-yBiFOqFg9SKod1vGrhd2d79I6cBeqWrEwBbn3SeFVqX6QptGYeRDMHaXfWCk6iT2'], 'uuids': ['c5f566c4-6058-4ef9-81db-bd73cd9b9965']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.102637", "end": "2020-06-04 08:21:53.228872", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-yBiFOqFg9SKod1vGrhd2d79I6cBeqWrEwBbn3SeFVqX6QptGYeRDMHaXfWCk6iT2"], "labels": [], "masters": [], "uuids": ["c5f566c4-6058-4ef9-81db-bd73cd9b9965"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:53.126235", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0985393 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0985393 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2020-06-04T08:21:52.949 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:52 +0000 (0:00:00.169) 0:02:26.416 ********* 2020-06-04T08:21:53.248 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-06-04T08:21:53.250 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.127235", "end": "2020-06-04 08:21:53.576402", "rc": 0, "start": "2020-06-04 08:21:53.449167", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-04T08:21:53.286 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:53 +0000 (0:00:00.336) 0:02:26.753 ********* 2020-06-04T08:21:53.446 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** changed: [smithi196.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-ypiX2xSrkSioOqIB7uUWqfvpyTEo69gpRSu58XrMhRo3DMa7NUpvVivlA9iTGLrS'], 'uuids': ['eac3d426-0ec1-42fd-a1ec-7e9fbe89193f']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.462843", "end": "2020-06-04 08:21:53.401857", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-ypiX2xSrkSioOqIB7uUWqfvpyTEo69gpRSu58XrMhRo3DMa7NUpvVivlA9iTGLrS"], "labels": [], "masters": [], "uuids": ["eac3d426-0ec1-42fd-a1ec-7e9fbe89193f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:52.939014", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.145882 s, 791 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.145882 s, 791 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-04T08:21:53.471 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:53.475 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:53.506 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-06-04T08:21:53.508 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.046262", "end": "2020-06-04 08:21:53.835603", "rc": 0, "start": "2020-06-04 08:21:53.789341", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-04T08:21:53.542 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:53 +0000 (0:00:00.256) 0:02:27.010 ********* 2020-06-04T08:21:53.582 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-06-04T08:21:53.624 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:53 +0000 (0:00:00.081) 0:02:27.091 ********* 2020-06-04T08:21:53.678 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:53.708 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:53 +0000 (0:00:00.084) 0:02:27.175 ********* 2020-06-04T08:21:53.749 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-06-04T08:21:53.751 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:53.791 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:53 +0000 (0:00:00.083) 0:02:27.259 ********* 2020-06-04T08:21:53.805 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-06-04T08:21:53.807 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-ypiX2xSrkSioOqIB7uUWqfvpyTEo69gpKcIBwQZldhFNI9ayVFMq4YmE0XVfhM7Q'], 'uuids': ['ad89084a-5c57-4de8-9f68-48d9fe090feb']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.154355", "end": "2020-06-04 08:21:53.718522", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-ypiX2xSrkSioOqIB7uUWqfvpyTEo69gpKcIBwQZldhFNI9ayVFMq4YmE0XVfhM7Q"], "labels": [], "masters": [], "uuids": ["ad89084a-5c57-4de8-9f68-48d9fe090feb"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:53.564167", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.149659 s, 771 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.149659 s, 771 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-04T08:21:53.808 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XGPQ', 'wwn-0x5000c50092928bab'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': '81cc74cc-5dd2-4c99-9728-93a5b2dec7d8', 'links': {'masters': [], 'labels': [ 2020-06-04T08:21:53.809 INFO:teuthology.task.ansible.out:], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XGPQ-part1', 'wwn-0x5000c50092928bab-part1'], 'uuids': ['81cc74cc-5dd2-4c99-9728-93a5b2dec7d8']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50092928bab', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5XGPQ', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XGPQ", "wwn-0x5000c50092928bab"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XGPQ-part1", "wwn-0x5000c50092928bab-part1"], "labels": [], "masters": [], "uuids": ["81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XGPQ", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092928bab"}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:53.839 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:53.888 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:53 +0000 (0:00:00.091) 0:02:27.351 ********* 2020-06-04T08:21:53.936 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:53.975 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:53 +0000 (0:00:00.091) 0:02:27.442 ********* 2020-06-04T08:21:53.985 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-06-04T08:21:53.986 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-EYXAoB-5E1w-suAF-KqgK-kTRk-r3xZ-govjsB', 'nvme-INTEL_SSDPEDMD400G4_CVFT6246000Z400BGN', 'nvme-nvme.8086-43564654363234363030305a34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_4', 'vg_nvme-lv_1', 'vg_nvme-lv_3', 'vg_nvme-lv_5', 'vg_nvme-lv_2'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.051412", "end": "2020-06-04 08:21:53.908969", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-EYXAoB-5E1w-suAF-KqgK-kTRk-r3xZ-govjsB", "nvme-INTEL_SSDPEDMD400G4_CVFT6246000Z400BGN", "nvme-nvme.8086-43564654363234363030305a34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:53.857557", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0467772 s, 2.5 GB/s", "std 2020-06-04T08:21:53.989 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0467772 s, 2.5 GB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi196.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:53.990 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:54.138 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-06-04T08:21:54.140 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-04T08:21:54.178 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:54 +0000 (0:00:00.203) 0:02:27.646 ********* 2020-06-04T08:21:54.237 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-06-04T08:21:54.238 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'block_used': 13741041, 'uuid': '81cc74cc-5dd2-4c99-9728-93a5b2dec7d8', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226354067, 'size_available': 927146258432, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149294, 'block_size': 4096, 'inode_available': 60905682}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226354067, "block_size": 4096, "block_total": 240095108, "block_used": 13741041, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905682, "inode_total": 61054976, "inode_used": 149294, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927146258432, "size_total": 983429562368, "uuid": "81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:54.276 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:54 +0000 (0:00:00.097) 0:02:27.744 ********* 2020-06-04T08:21:54.295 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** changed: [smithi196.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-ypiX2xSrkSioOqIB7uUWqfvpyTEo69gpTDR85VraZx8DK1SIn50BBLLWDHM0l4RI'], 'uuids': ['ba1a4c9b-c2aa-4299-8f6c-391001fd94f5']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.152770", "end": "2020-06-04 08:21:54.213658", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-ypiX2xSrkSioOqIB7uUWqfvpyTEo69gpTDR85VraZx8DK1SIn50BBLLWDHM0l4RI"], "labels": [], "masters": [], "uuids": ["ba1a4c9b-c2aa-4299-8f6c-391001fd94f5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:54.060888", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.149606 s, 771 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.149606 s, 771 MB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi196.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': 2020-06-04T08:21:54.297 INFO:teuthology.task.ansible.out:'0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:54.330 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:54.369 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:54 +0000 (0:00:00.093) 0:02:27.837 ********* 2020-06-04T08:21:54.413 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-04T08:21:54.415 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:54.452 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:54 +0000 (0:00:00.082) 0:02:27.919 ********* 2020-06-04T08:21:54.514 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:54.560 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:54 +0000 (0:00:00.107) 0:02:28.027 ********* 2020-06-04T08:21:54.580 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-06-04T08:21:54.582 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-ypiX2xSrkSioOqIB7uUWqfvpyTEo69gpC5ByS5Qtm5Zy4fJUBQ1uMl75GBA02LBE'], 'uuids': ['9baaeb27-ce36-42b8-9327-8b9b6a424350']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.155610", "end": "2020-06-04 08:21:54.523809", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-ypiX2xSrkSioOqIB7uUWqfvpyTEo69gpC5ByS5Qtm5Zy4fJUBQ1uMl75GBA02LBE"], "labels": [], "masters": [], "uuids": ["9baaeb27-ce36-42b8-9327-8b9b6a424350"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:54.368199", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151048 s, 764 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151048 s, 764 MB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi196.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': 2020-06-04T08:21:54.584 INFO:teuthology.task.ansible.out: '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:54.604 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:54.660 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:54 +0000 (0:00:00.087) 0:02:28.114 ********* 2020-06-04T08:21:54.700 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-06-04T08:21:54.702 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:54.744 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:54 +0000 (0:00:00.097) 0:02:28.211 ********* 2020-06-04T08:21:54.793 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:54.838 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:54 +0000 (0:00:00.084) 0:02:28.296 ********* 2020-06-04T08:21:54.860 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** changed: [smithi196.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-ypiX2xSrkSioOqIB7uUWqfvpyTEo69gpDrvDblB3T0XJMFr6yJSZs5cfmLZbFQoK'], 'uuids': ['9765944d-02e4-408c-a822-1a32035891e9']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.154048", "end": "2020-06-04 08:21:54.820075", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-ypiX2xSrkSioOqIB7uUWqfvpyTEo69gpDrvDblB3T0XJMFr6yJSZs5cfmLZbFQoK"], "labels": [], "masters": [], "uuids": ["9765944d-02e4-408c-a822-1a32035891e9"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-06-04 08:21:54.666027", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.149302 s, 773 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.149302 s, 773 MB/s"], "stdout": "", "stdout_lines": []} 2020-06-04T08:21:54.910 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:54 +0000 (0:00:00.077) 0:02:28.374 ********* 2020-06-04T08:21:55.215 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi196.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.134347", "end": "2020-06-04 08:21:55.168195", "rc": 0, "start": "2020-06-04 08:21:55.033848", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-04T08:21:55.256 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:55 +0000 (0:00:00.350) 0:02:28.724 ********* 2020-06-04T08:21:55.470 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-06-04T08:21:55.473 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.058923", "end": "2020-06-04 08:21:55.430618", "rc": 0, "start": "2020-06-04 08:21:55.371695", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-04T08:21:55.507 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:55 +0000 (0:00:00.251) 0:02:28.975 ********* 2020-06-04T08:21:55.519 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-06-04T08:21:55.579 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:55 +0000 (0:00:00.071) 0:02:29.047 ********* 2020-06-04T08:21:55.591 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-06-04T08:21:55.636 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:55 +0000 (0:00:00.056) 0:02:29.103 ********* 2020-06-04T08:21:55.687 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-06-04T08:21:55.688 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:55.738 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:55 +0000 (0:00:00.101) 0:02:29.205 ********* 2020-06-04T08:21:55.780 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-06-04T08:21:55.782 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:55.829 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:55 +0000 (0:00:00.089) 0:02:29.295 ********* 2020-06-04T08:21:55.872 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:55.928 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:55 +0000 (0:00:00.099) 0:02:29.395 ********* 2020-06-04T08:21:55.995 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:56.050 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:56 +0000 (0:00:00.121) 0:02:29.517 ********* 2020-06-04T08:21:56.202 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi196.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-06-04T08:21:56.240 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:56 +0000 (0:00:00.190) 0:02:29.707 ********* 2020-06-04T08:21:56.249 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-06-04T08:21:56.251 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-04T08:21:56.303 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-06-04T08:21:56.323 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'block_used': 13741035, 'uuid': '81cc74cc-5dd2-4c99-9728-93a5b2dec7d8', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226354073, 'size_available': 927146283008, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149293, 'block_size': 4096, 'inode_available': 60905683}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226354073, "block_size": 4096, "block_total": 240095108, "block_used": 13741035, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905683, "inode_total": 61054976, "inode_used": 149293, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927146283008, "size_total": 983429562368, "uuid": "81cc74cc-5dd2-4c99-9728-93a5b2dec7d8"}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:56.349 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:56 +0000 (0:00:00.109) 0:02:29.817 ********* 2020-06-04T08:21:56.398 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:56.474 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:56 +0000 (0:00:00.119) 0:02:29.936 ********* 2020-06-04T08:21:56.484 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-04T08:21:56.509 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:56.559 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:56 +0000 (0:00:00.084) 0:02:30.022 ********* 2020-06-04T08:21:56.603 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:56.645 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:56 +0000 (0:00:00.089) 0:02:30.111 ********* 2020-06-04T08:21:56.664 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-06-04T08:21:56.666 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-04T08:21:56.675 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-06-04T08:21:56.677 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:56.721 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:56 +0000 (0:00:00.077) 0:02:30.188 ********* 2020-06-04T08:21:56.777 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:56.824 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:56 +0000 (0:00:00.103) 0:02:30.292 ********* 2020-06-04T08:21:56.877 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:56.909 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:56 +0000 (0:00:00.084) 0:02:30.377 ********* 2020-06-04T08:21:56.921 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-06-04T08:21:56.925 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-04T08:21:57.138 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-04T08:21:57.180 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:57 +0000 (0:00:00.270) 0:02:30.647 ********* 2020-06-04T08:21:57.362 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-06-04T08:21:57.364 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-06-04T08:21:57.414 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:57 +0000 (0:00:00.234) 0:02:30.881 ********* 2020-06-04T08:21:57.455 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-06-04T08:21:57.457 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-06-04T08:21:57.496 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:57 +0000 (0:00:00.082) 0:02:30.964 ********* 2020-06-04T08:21:57.607 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-06-04T08:21:57.610 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-04T08:21:57.744 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-04T08:21:57.829 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-06-04T08:21:57.831 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-04T08:21:57.871 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-06-04T08:21:57.873 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-04T08:21:58.007 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-04T08:21:58.019 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:58.058 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:58 +0000 (0:00:00.562) 0:02:31.526 ********* 2020-06-04T08:21:58.102 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-06-04T08:21:58.104 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-04T08:21:58.212 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-06-04T08:21:58.214 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-06-04T08:21:58.249 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:58 +0000 (0:00:00.190) 0:02:31.717 ********* 2020-06-04T08:21:58.322 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-06-04T08:21:58.324 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} TASK [testnode : Create logical volume(s)] ************************************* 2020-06-04T08:21:58.325 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-04T08:21:58.362 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:58 +0000 (0:00:00.113) 0:02:31.830 ********* 2020-06-04T08:21:58.558 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-04T08:21:58.822 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-06-04T08:21:58.860 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:58 +0000 (0:00:00.497) 0:02:32.328 ********* 2020-06-04T08:21:58.893 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-06-04T08:21:58.894 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true} 2020-06-04T08:21:58.960 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:58 +0000 (0:00:00.100) 0:02:32.428 ********* 2020-06-04T08:21:59.024 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-06-04T08:21:59.025 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-06-04T08:21:59.070 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:59 +0000 (0:00:00.109) 0:02:32.538 ********* 2020-06-04T08:21:59.250 INFO:teuthology.task.ansible.out: 2020-06-04T08:21:59.252 INFO:teuthology.task.ansible.out:TASK [testnode : Write /scratch_devs] ****************************************** changed: [smithi196.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-04T08:21:59.406 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-04T08:21:59.505 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-06-04T08:21:59.507 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-06-04T08:21:59.558 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:59 +0000 (0:00:00.484) 0:02:33.022 ********* 2020-06-04T08:21:59.566 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-06-04T08:21:59.567 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-04T08:21:59.667 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-06-04T08:21:59.690 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-06-04T08:21:59.732 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:59 +0000 (0:00:00.178) 0:02:33.200 ********* 2020-06-04T08:21:59.743 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-06-04T08:21:59.745 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-06-04T08:21:59.783 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:59 +0000 (0:00:00.050) 0:02:33.251 ********* 2020-06-04T08:21:59.873 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-06-04T08:21:59.918 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:59 +0000 (0:00:00.135) 0:02:33.386 ********* 2020-06-04T08:21:59.950 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-06-04T08:21:59.951 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-06-04T08:21:59.991 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:21:59 +0000 (0:00:00.071) 0:02:33.458 ********* 2020-06-04T08:22:00.006 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-06-04T08:22:00.008 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-06-04T08:22:00.052 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:00 +0000 (0:00:00.061) 0:02:33.519 ********* 2020-06-04T08:22:00.320 INFO:teuthology.task.ansible.out: 2020-06-04T08:22:00.321 INFO:teuthology.task.ansible.out:TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi196.front.sepia.ceph.com] => {"changed": true} 2020-06-04T08:22:00.381 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:00 +0000 (0:00:00.328) 0:02:33.848 ********* 2020-06-04T08:22:00.388 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-06-04T08:22:00.390 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1215, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-06-04T08:22:00.430 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:00 +0000 (0:00:00.049) 0:02:33.898 ********* 2020-06-04T08:22:00.597 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-06-04T08:22:00.599 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-06-04T08:22:00.663 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:00 +0000 (0:00:00.232) 0:02:34.131 ********* 2020-06-04T08:22:00.844 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-06-04T08:22:00.883 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:00 +0000 (0:00:00.219) 0:02:34.350 ********* 2020-06-04T08:22:00.892 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-06-04T08:22:00.894 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-06-04T08:22:00.942 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:00 +0000 (0:00:00.059) 0:02:34.410 ********* 2020-06-04T08:22:01.034 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-06-04T08:22:01.036 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-06-04T08:22:01.088 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:01 +0000 (0:00:00.145) 0:02:34.555 ********* 2020-06-04T08:22:01.471 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1215, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-06-04T08:22:01.528 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:01 +0000 (0:00:00.439) 0:02:34.995 ********* 2020-06-04T08:22:01.920 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1591258834, "cache_updated": false, "changed": false} TASK [testnode : Upload CPAN config for root.] ********************************* ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-06-04T08:22:01.963 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:01 +0000 (0:00:00.433) 0:02:35.429 ********* 2020-06-04T08:22:02.005 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:01 +0000 (0:00:00.043) 0:02:35.472 ********* 2020-06-04T08:22:02.164 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-06-04T08:22:02.166 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.047714", "end": "2020-06-04 08:22:02.511068", "rc": 0, "start": "2020-06-04 08:22:02.463354", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-06-04T08:22:02.209 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:02 +0000 (0:00:00.203) 0:02:35.676 ********* 2020-06-04T08:22:02.273 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:02.322 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:02 +0000 (0:00:00.105) 0:02:35.782 ********* 2020-06-04T08:22:02.347 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:02.401 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:02 +0000 (0:00:00.084) 0:02:35.866 ********* 2020-06-04T08:22:02.893 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-06-04T08:22:02.895 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"cache_update_time": 1591258833, "cache_updated": false, "changed": false} 2020-06-04T08:22:02.946 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:02 +0000 (0:00:00.544) 0:02:36.411 ********* 2020-06-04T08:22:03.147 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-06-04T08:22:03.359 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.041440", "end": "2020-06-04 08:22:03.109368", "rc": 0, "start": "2020-06-04 08:22:03.067928", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} Thursday 04 June 2020 08:22:03 +0000 (0:00:00.240) 0:02:36.651 ********* TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 04 June 2020 08:22:03 +0000 (0:00:00.077) 0:02:36.729 ********* TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Install ntp package on apt based systems.] ******************** ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1591258834, "cache_updated": false, "changed": false} Thursday 04 June 2020 08:22:03 +0000 (0:00:00.093) 0:02:36.822 ********* 2020-06-04T08:22:03.395 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:03 +0000 (0:00:00.040) 0:02:36.862 ********* 2020-06-04T08:22:03.405 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-06-04T08:22:03.407 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:03.452 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:03 +0000 (0:00:00.056) 0:02:36.919 ********* 2020-06-04T08:22:03.828 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-06-04T08:22:03.830 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-06-04T08:22:03.873 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:03 +0000 (0:00:00.421) 0:02:37.341 ********* 2020-06-04T08:22:03.920 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-06-04T08:22:03.922 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:03.959 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:03 +0000 (0:00:00.085) 0:02:37.427 ********* 2020-06-04T08:22:04.176 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-06-04T08:22:04.181 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-06-04 08:18:56 UTC", "ActiveEnterTimestampMonotonic": "25770741", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service system.slice basic.target network.target systemd-journald.socket -.mount sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-06-04 08:18:49 UTC", "AssertTimestampMonotonic": "19220127", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-06-04 08:18:49 UTC", "ConditionTimestampMonotonic": "19220127", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-06-04T08:22:04.183 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1760", "ExecMainStartTimestamp": "Thu 2020-06-04 08:18:56 UTC", "ExecMainStartTimestampMonotonic": "25770691", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-06-04 08:18:49 UTC", "InactiveExitTimestampMonotonic": "19220836", "InvocationID": "9f64f27669ce4ef882e000fd91d2c38d", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-06-04T08:22:04.185 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1760", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-06-04 08:18:56 UTC", "S 2020-06-04T08:22:04.187 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "25770741", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-06-04 08:18:56 UTC", "WatchdogTimestampMonotonic": "25770738", "WatchdogUSec": "0"}} 2020-06-04T08:22:04.236 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:04 +0000 (0:00:00.277) 0:02:37.704 ********* 2020-06-04T08:22:04.261 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-06-04T08:22:04.263 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"cache_update_time": 1591258833, "cache_updated": false, "changed": false} 2020-06-04T08:22:04.312 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:04 +0000 (0:00:00.075) 0:02:37.779 ********* 2020-06-04T08:22:04.329 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-04T08:22:04.331 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-06-04T08:22:04.379 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:04 +0000 (0:00:00.067) 0:02:37.847 ********* 2020-06-04T08:22:04.389 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-06-04T08:22:04.390 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:04.438 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:04 +0000 (0:00:00.058) 0:02:37.905 ********* 2020-06-04T08:22:04.449 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-04T08:22:04.451 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:04.489 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:04 +0000 (0:00:00.051) 0:02:37.957 ********* 2020-06-04T08:22:04.528 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-06-04T08:22:04.531 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:04.568 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:04 +0000 (0:00:00.079) 0:02:38.036 ********* 2020-06-04T08:22:04.811 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-06-04T08:22:04.879 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:04 +0000 (0:00:00.309) 0:02:38.345 ********* 2020-06-04T08:22:04.929 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-06-04T08:22:04.931 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:04.971 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:04 +0000 (0:00:00.090) 0:02:38.436 ********* 2020-06-04T08:22:05.187 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2020-06-04 08:19:16 UTC", "ActiveEnterTimestampMonotonic": "24415456", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket system.slice basic.target sysinit.target systemd-tmpfiles-setup.service network.target -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Thu 2020-06-04 08:19:09 UTC", "AssertTimestampMonotonic": "17872064", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2020-06-04 08:19:09 UTC", "ConditionTimestampMonotonic": "17872064", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Dele 2020-06-04T08:22:05.189 INFO:teuthology.task.ansible.out:gate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documentation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1858", "ExecMainStartTimestamp": "Thu 2020-06-04 08:19:16 UTC", "ExecMainStartTimestampMonotonic": "24415396", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2020-06-04 08:19:09 UTC", "InactiveExitTimestampMonotonic": "17872806", "InvocationID": "05392b36139e427087dd7077da546686", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128031", "LimitNPROCSoft": "128031", "LimitRSS": "infi 2020-06-04T08:22:05.193 INFO:teuthology.task.ansible.out:nity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128031", "LimitSIGPENDINGSoft": "128031", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1858", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "Star 2020-06-04T08:22:05.195 INFO:teuthology.task.ansible.out:tupIOWeight": "[not set]", "StateChangeTimestamp": "Thu 2020-06-04 08:19:16 UTC", "StateChangeTimestampMonotonic": "24415456", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Thu 2020-06-04 08:19:16 UTC", "WatchdogTimestampMonotonic": "24415452", "WatchdogUSec": "0"}} 2020-06-04T08:22:05.231 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:05 +0000 (0:00:00.262) 0:02:38.699 ********* 2020-06-04T08:22:05.260 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi196.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-06-04T08:22:05.299 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:05 +0000 (0:00:00.067) 0:02:38.766 ********* 2020-06-04T08:22:05.327 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-04T08:22:05.328 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:05.370 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:05 +0000 (0:00:00.071) 0:02:38.837 ********* 2020-06-04T08:22:05.411 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:05.448 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:05 +0000 (0:00:00.077) 0:02:38.915 ********* 2020-06-04T08:22:05.569 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-06-04T08:22:05.571 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1591258834, "cache_updated": false, "changed": false} 2020-06-04T08:22:05.629 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:05 +0000 (0:00:00.182) 0:02:39.097 ********* 2020-06-04T08:22:05.663 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-06-04T08:22:05.665 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:05.702 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:05 +0000 (0:00:00.072) 0:02:39.170 ********* 2020-06-04T08:22:05.862 INFO:teuthology.task.ansible.out: 2020-06-04T08:22:05.864 INFO:teuthology.task.ansible.out:TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-06-04T08:22:05.902 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:05 +0000 (0:00:00.200) 0:02:39.370 ********* 2020-06-04T08:22:06.317 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-06-04T08:22:06.319 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"cache_update_time": 1591258833, "cache_updated": false, "changed": false} 2020-06-04T08:22:06.361 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:06 +0000 (0:00:00.459) 0:02:39.829 ********* 2020-06-04T08:22:06.380 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-06-04T08:22:06.382 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-06-04T08:22:06.439 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:06 +0000 (0:00:00.067) 0:02:39.896 ********* TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:06.478 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:06 +0000 (0:00:00.049) 0:02:39.945 ********* 2020-06-04T08:22:06.638 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-06-04T08:22:06.676 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:06 +0000 (0:00:00.198) 0:02:40.144 ********* 2020-06-04T08:22:07.073 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-06-04T08:22:07.075 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-06-04T08:22:07.114 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:07 +0000 (0:00:00.437) 0:02:40.581 ********* 2020-06-04T08:22:07.518 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-06-04T08:22:07.520 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-06-04T08:22:07.557 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:07 +0000 (0:00:00.442) 0:02:41.024 ********* 2020-06-04T08:22:07.915 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-06-04T08:22:07.967 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:07 +0000 (0:00:00.409) 0:02:41.434 ********* 2020-06-04T08:22:08.093 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-06-04T08:22:08.095 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false} 2020-06-04T08:22:08.141 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:08 +0000 (0:00:00.174) 0:02:41.608 ********* 2020-06-04T08:22:08.183 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2020-06-04T08:22:08.220 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:08 +0000 (0:00:00.077) 0:02:41.686 ********* 2020-06-04T08:22:08.403 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-06-04T08:22:08.405 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1591258500.856294, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1591258500.852294, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 10034, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1591258500.852294, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "3519411719", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-06-04T08:22:08.446 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:08 +0000 (0:00:00.227) 0:02:41.913 ********* 2020-06-04T08:22:08.487 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-06-04T08:22:08.489 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false} 2020-06-04T08:22:08.531 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:08 +0000 (0:00:00.079) 0:02:41.993 ********* 2020-06-04T08:22:08.568 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ ok: [smithi196.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-06-04T08:22:08.603 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:08 +0000 (0:00:00.077) 0:02:42.070 ********* 2020-06-04T08:22:08.904 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-06-04T08:22:08.906 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-06-04T08:22:09.061 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:09 +0000 (0:00:00.458) 0:02:42.529 ********* 2020-06-04T08:22:09.073 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-06-04T08:22:09.076 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1591258536.9573047, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1591258535.4900386, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 10034, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1591258535.4900386, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3519411719", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-06-04T08:22:09.122 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:09 +0000 (0:00:00.061) 0:02:42.590 ********* 2020-06-04T08:22:09.134 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-06-04T08:22:09.135 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-06-04T08:22:09.173 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:09 +0000 (0:00:00.050) 0:02:42.640 ********* 2020-06-04T08:22:09.212 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-06-04T08:22:09.214 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:09.254 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:09 +0000 (0:00:00.081) 0:02:42.721 ********* 2020-06-04T08:22:09.274 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-06-04T08:22:09.276 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-06-04T08:22:09.315 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:09 +0000 (0:00:00.060) 0:02:42.782 ********* 2020-06-04T08:22:09.376 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [smithi196.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-06-04T08:22:09.414 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:09 +0000 (0:00:00.098) 0:02:42.881 ********* 2020-06-04T08:22:09.422 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-06-04T08:22:09.424 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-06-04T08:22:09.468 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:09 +0000 (0:00:00.054) 0:02:42.936 ********* 2020-06-04T08:22:09.479 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-06-04T08:22:09.482 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:09.527 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:09 +0000 (0:00:00.058) 0:02:42.995 ********* 2020-06-04T08:22:09.612 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-06-04T08:22:09.615 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-06-04T08:22:09.671 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:09 +0000 (0:00:00.143) 0:02:43.138 ********* 2020-06-04T08:22:09.693 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-06-04T08:22:09.695 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-06-04T08:22:09.745 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:09 +0000 (0:00:00.073) 0:02:43.211 ********* 2020-06-04T08:22:09.753 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:09.797 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:09 +0000 (0:00:00.053) 0:02:43.265 ********* 2020-06-04T08:22:09.851 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi060.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2020-06-04T08:22:09.854 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2020-06-04T08:22:09.863 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-06-04T08:22:09.865 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-06-04T08:22:09.900 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:09 +0000 (0:00:00.102) 0:02:43.368 ********* 2020-06-04T08:22:09.909 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-06-04T08:22:09.911 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-06-04T08:22:09.947 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:09 +0000 (0:00:00.046) 0:02:43.414 ********* 2020-06-04T08:22:09.957 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-06-04T08:22:09.959 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:09.960 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:09.961 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:09.975 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.013 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:10 +0000 (0:00:00.066) 0:02:43.481 ********* 2020-06-04T08:22:10.023 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.068 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:10 +0000 (0:00:00.054) 0:02:43.536 ********* 2020-06-04T08:22:10.077 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-04T08:22:10.081 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.121 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:10 +0000 (0:00:00.052) 0:02:43.589 ********* 2020-06-04T08:22:10.131 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-06-04T08:22:10.133 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.148 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.151 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.190 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:10 +0000 (0:00:00.068) 0:02:43.657 ********* 2020-06-04T08:22:10.199 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-06-04T08:22:10.201 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.243 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:10 +0000 (0:00:00.053) 0:02:43.711 ********* 2020-06-04T08:22:10.257 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-06-04T08:22:10.259 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.260 INFO:teuthology.task.ansible.out: skipping: [smithi196.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.262 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi196.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.264 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.300 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:10 +0000 (0:00:00.056) 0:02:43.767 ********* 2020-06-04T08:22:10.301 INFO:teuthology.task.ansible.out: 2020-06-04T08:22:10.310 INFO:teuthology.task.ansible.out: 2020-06-04T08:22:10.312 INFO:teuthology.task.ansible.out:TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.351 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:10 +0000 (0:00:00.051) 0:02:43.818 ********* 2020-06-04T08:22:10.353 INFO:teuthology.task.ansible.out: 2020-06-04T08:22:10.361 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-06-04T08:22:10.362 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.406 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:10 +0000 (0:00:00.055) 0:02:43.874 ********* 2020-06-04T08:22:10.415 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-06-04T08:22:10.417 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.454 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:10 +0000 (0:00:00.047) 0:02:43.922 ********* 2020-06-04T08:22:10.462 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-06-04T08:22:10.464 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.505 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:10 +0000 (0:00:00.051) 0:02:43.973 ********* 2020-06-04T08:22:10.514 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-06-04T08:22:10.516 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.555 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:10 +0000 (0:00:00.050) 0:02:44.023 ********* 2020-06-04T08:22:10.565 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-06-04T08:22:10.567 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.607 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:10 +0000 (0:00:00.051) 0:02:44.074 ********* 2020-06-04T08:22:10.616 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-06-04T08:22:10.618 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.661 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:10 +0000 (0:00:00.054) 0:02:44.129 ********* 2020-06-04T08:22:10.671 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-06-04T08:22:10.673 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.709 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:10 +0000 (0:00:00.047) 0:02:44.177 ********* 2020-06-04T08:22:10.720 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-06-04T08:22:10.721 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.765 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:10 +0000 (0:00:00.055) 0:02:44.232 ********* 2020-06-04T08:22:10.773 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-06-04T08:22:10.774 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.818 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:10 +0000 (0:00:00.053) 0:02:44.285 ********* 2020-06-04T08:22:10.827 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-06-04T08:22:10.829 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.867 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:10 +0000 (0:00:00.048) 0:02:44.334 ********* 2020-06-04T08:22:10.876 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-06-04T08:22:10.877 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.916 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:10 +0000 (0:00:00.049) 0:02:44.384 ********* 2020-06-04T08:22:10.957 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-06-04T08:22:10.959 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-06-04T08:22:10.997 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:10 +0000 (0:00:00.080) 0:02:44.464 ********* 2020-06-04T08:22:11.047 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-06-04T08:22:11.049 INFO:teuthology.task.ansible.out:skipping: [smithi196.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi060.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011193", "end": "2020-06-04 08:22:11.363654", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-06-04 08:22:11.352461", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-04T08:22:11.090 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:11 +0000 (0:00:00.092) 0:02:44.557 ********* 2020-06-04T08:22:11.161 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.004175", "end": "2020-06-04 08:22:11.493516", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-06-04 08:22:11.489341", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-04T08:22:11.278 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010182", "end": "2020-06-04 08:22:11.233654", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-06-04 08:22:11.223472", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-04T08:22:11.347 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005454", "end": "2020-06-04 08:22:11.647600", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-06-04 08:22:11.642146", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-06-04T08:22:11.430 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.005899", "end": "2020-06-04 08:22:11.385925", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-06-04 08:22:11.380026", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-04T08:22:11.496 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.022373", "end": "2020-06-04 08:22:11.816302", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-06-04 08:22:11.793929", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-04T08:22:11.536 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:11 +0000 (0:00:00.446) 0:02:45.004 ********* 2020-06-04T08:22:11.569 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005631", "end": "2020-06-04 08:22:11.529454", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-06-04 08:22:11.523823", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-06-04T08:22:11.592 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-06-04T08:22:11.637 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:11 +0000 (0:00:00.100) 0:02:45.104 ********* 2020-06-04T08:22:11.639 INFO:teuthology.task.ansible.out: 2020-06-04T08:22:11.815 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-06-04T08:22:11.902 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-06-04T08:22:11.903 INFO:teuthology.task.ansible.out:ok: [smithi196.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.158785", "end": "2020-06-04 08:22:11.830016", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-06-04 08:22:11.671231", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-06-04T08:22:11.938 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:11 +0000 (0:00:00.300) 0:02:45.405 ********* 2020-06-04T08:22:11.978 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-06-04T08:22:12.021 INFO:teuthology.task.ansible.out:Thursday 04 June 2020 08:22:12 +0000 (0:00:00.083) 0:02:45.488 ********* 2020-06-04T08:22:12.172 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-06-04T08:22:12.174 INFO:teuthology.task.ansible.out:changed: [smithi196.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-06-04T08:22:12.294 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** 2020-06-04T08:22:12.296 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2020-06-04T08:22:12.297 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-06-04T08:22:12.299 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** 2020-06-04T08:22:12.301 INFO:teuthology.task.ansible.out:skipping: no hosts matched PLAY RECAP ********************************************************************* smithi060.front.sepia.ceph.com : ok=114 changed=34 unreachable=0 failed=0 skipped=175 rescued=0 ignored=0 smithi196.front.sepia.ceph.com : ok=114 changed=34 unreachable=0 failed=0 skipped=175 rescued=0 ignored=0 Thursday 04 June 2020 08:22:12 +0000 (0:00:00.279) 0:02:45.768 ********* =============================================================================== users : Create all admin users with sudo access. ----------------------- 20.42s users : Update authorized_keys using the keys repo --------------------- 18.80s testnode : Ensure packages are not present. ----------------------------- 9.14s testnode : Upgrade packages --------------------------------------------- 8.61s testnode : Install packages --------------------------------------------- 7.86s testnode : Zap all non-root disks --------------------------------------- 7.09s users : Remove revoked users -------------------------------------------- 5.55s common : Update apt cache ----------------------------------------------- 4.24s common : Install nrpe package and dependencies (Ubuntu/Fedora) ---------- 2.79s testnode : Update apt cache. -------------------------------------------- 2.67s testnode : Install apt keys --------------------------------------------- 2.30s Gathering Facts --------------------------------------------------------- 1.97s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 1.17s users : Update authorized_keys for each user with literal keys ---------- 1.16s testnode : Set the system's hostname ------------------------------------ 1.02s ansible-managed : Add authorized keys for the ansible user. ------------- 0.94s testnode : Install python-apt ------------------------------------------- 0.88s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 0.86s Gather 2020-06-04T08:22:12.308 INFO:teuthology.task.ansible.out:ing Facts --------------------------------------------------------- 0.82s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 0.77s 2020-06-04T08:22:12.667 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi060.front.sepia.ceph.com'), Remote(name='ubuntu@smithi196.front.sepia.ceph.com')] 2020-06-04T08:22:12.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-04T08:22:12.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:22:12.751 INFO:teuthology.orchestra.run.smithi060:> true 2020-06-04T08:22:12.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-06-04T08:22:12.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-06-04T08:22:13.039 INFO:teuthology.orchestra.run.smithi196:> true 2020-06-04T08:22:13.241 INFO:teuthology.run_tasks:Running task clock... 2020-06-04T08:22:13.257 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-06-04T08:22:13.258 INFO:teuthology.orchestra.run:Running command with timeout 60 2020-06-04T08:22:13.259 INFO:teuthology.orchestra.run.smithi060:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-06-04T08:22:13.316 INFO:teuthology.orchestra.run.smithi060.stdout: 4 Jun 08:22:13 ntpd[8810]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-06-04T08:22:13.317 INFO:teuthology.orchestra.run.smithi060.stdout: 4 Jun 08:22:13 ntpd[8810]: Command line: ntpd -gq 2020-06-04T08:22:13.321 INFO:teuthology.orchestra.run.smithi060.stdout: 4 Jun 08:22:13 ntpd[8810]: proto: precision = 0.126 usec (-23) 2020-06-04T08:22:13.321 INFO:teuthology.orchestra.run.smithi060.stdout: 4 Jun 08:22:13 ntpd[8810]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-06-04T08:22:13.322 INFO:teuthology.orchestra.run.smithi060.stdout: 4 Jun 08:22:13 ntpd[8810]: restrict ::: KOD does nothing without LIMITED. 2020-06-04T08:22:13.322 INFO:teuthology.orchestra.run.smithi060.stdout: 4 Jun 08:22:13 ntpd[8810]: Listen and drop on 0 v6wildcard [::]:123 2020-06-04T08:22:13.323 INFO:teuthology.orchestra.run.smithi060.stdout: 4 Jun 08:22:13 ntpd[8810]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-06-04T08:22:13.323 INFO:teuthology.orchestra.run.smithi060.stdout: 4 Jun 08:22:13 ntpd[8810]: Listen normally on 2 lo 127.0.0.1:123 2020-06-04T08:22:13.323 INFO:teuthology.orchestra.run.smithi060.stdout: 4 Jun 08:22:13 ntpd[8810]: Listen normally on 3 ens1f0 172.21.15.60:123 2020-06-04T08:22:13.324 INFO:teuthology.orchestra.run.smithi060.stdout: 4 Jun 08:22:13 ntpd[8810]: Listen normally on 4 lo [::1]:123 2020-06-04T08:22:13.324 INFO:teuthology.orchestra.run.smithi060.stdout: 4 Jun 08:22:13 ntpd[8810]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1560%3]:123 2020-06-04T08:22:13.324 INFO:teuthology.orchestra.run.smithi060.stdout: 4 Jun 08:22:13 ntpd[8810]: Listening on routing socket on fd #22 for interface updates 2020-06-04T08:22:13.324 INFO:teuthology.orchestra.run.smithi060.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-06-04T08:22:13.325 INFO:teuthology.orchestra.run.smithi060.stderr:restrict ::: KOD does nothing without LIMITED. 2020-06-04T08:22:14.318 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:14 ntpd[8810]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-04T08:22:14.318 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:14 ntpd[8810]: can't open /var/log/ntpstats/rawstats.20200604: Permission denied 2020-06-04T08:22:14.319 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:14 ntpd[8810]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-04T08:22:14.319 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:14 ntpd[8810]: can't open /var/log/ntpstats/peerstats.20200604: Permission denied 2020-06-04T08:22:14.319 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:14 ntpd[8810]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-04T08:22:14.319 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:14 ntpd[8810]: can't open /var/log/ntpstats/rawstats.20200604: Permission denied 2020-06-04T08:22:14.320 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:14 ntpd[8810]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-04T08:22:14.320 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:14 ntpd[8810]: can't open /var/log/ntpstats/peerstats.20200604: Permission denied 2020-06-04T08:22:16.318 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:16 ntpd[8810]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-04T08:22:16.318 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:16 ntpd[8810]: can't open /var/log/ntpstats/rawstats.20200604: Permission denied 2020-06-04T08:22:16.319 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:16 ntpd[8810]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-04T08:22:16.319 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:16 ntpd[8810]: can't open /var/log/ntpstats/peerstats.20200604: Permission denied 2020-06-04T08:22:16.319 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:16 ntpd[8810]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-04T08:22:16.319 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:16 ntpd[8810]: can't open /var/log/ntpstats/rawstats.20200604: Permission denied 2020-06-04T08:22:16.319 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:16 ntpd[8810]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-04T08:22:16.320 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:16 ntpd[8810]: can't open /var/log/ntpstats/peerstats.20200604: Permission denied 2020-06-04T08:22:16.320 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:16 ntpd[8810]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-04T08:22:16.320 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:16 ntpd[8810]: can't open /var/log/ntpstats/rawstats.20200604: Permission denied 2020-06-04T08:22:16.320 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:16 ntpd[8810]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-04T08:22:16.320 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:16 ntpd[8810]: can't open /var/log/ntpstats/peerstats.20200604: Permission denied 2020-06-04T08:22:16.320 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:16 ntpd[8810]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-04T08:22:16.321 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:16 ntpd[8810]: can't open /var/log/ntpstats/rawstats.20200604: Permission denied 2020-06-04T08:22:16.321 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:16 ntpd[8810]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-04T08:22:16.321 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:16 ntpd[8810]: can't open /var/log/ntpstats/peerstats.20200604: Permission denied 2020-06-04T08:22:18.319 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:18 ntpd[8810]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-04T08:22:18.320 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:18 ntpd[8810]: can't open /var/log/ntpstats/rawstats.20200604: Permission denied 2020-06-04T08:22:18.320 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:18 ntpd[8810]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-04T08:22:18.320 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:18 ntpd[8810]: can't open /var/log/ntpstats/peerstats.20200604: Permission denied 2020-06-04T08:22:18.320 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:18 ntpd[8810]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-04T08:22:18.321 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:18 ntpd[8810]: can't open /var/log/ntpstats/rawstats.20200604: Permission denied 2020-06-04T08:22:18.321 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:18 ntpd[8810]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-04T08:22:18.321 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:18 ntpd[8810]: can't open /var/log/ntpstats/peerstats.20200604: Permission denied 2020-06-04T08:22:18.321 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:18 ntpd[8810]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-04T08:22:18.321 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:18 ntpd[8810]: can't open /var/log/ntpstats/rawstats.20200604: Permission denied 2020-06-04T08:22:18.322 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:18 ntpd[8810]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-04T08:22:18.322 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:18 ntpd[8810]: can't open /var/log/ntpstats/peerstats.20200604: Permission denied 2020-06-04T08:22:18.322 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:18 ntpd[8810]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-04T08:22:18.322 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:18 ntpd[8810]: can't open /var/log/ntpstats/rawstats.20200604: Permission denied 2020-06-04T08:22:18.322 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:18 ntpd[8810]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-04T08:22:18.323 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:18 ntpd[8810]: can't open /var/log/ntpstats/peerstats.20200604: Permission denied 2020-06-04T08:22:20.317 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:20 ntpd[8810]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-04T08:22:20.318 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:20 ntpd[8810]: can't open /var/log/ntpstats/rawstats.20200604: Permission denied 2020-06-04T08:22:20.318 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:20 ntpd[8810]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-04T08:22:20.318 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:20 ntpd[8810]: can't open /var/log/ntpstats/peerstats.20200604: Permission denied 2020-06-04T08:22:20.318 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:20 ntpd[8810]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-06-04T08:22:20.318 INFO:teuthology.orchestra.run.smithi060.stderr: 4 Jun 08:22:20 ntpd[8810]: can't open /var/log/ntpstats/loopstats.20200604: Permission denied 2020-06-04T08:22:20.319 INFO:teuthology.orchestra.run.smithi060.stdout: 4 Jun 08:22:20 ntpd[8810]: ntpd: time slew -0.001685 s 2020-06-04T08:22:20.319 INFO:teuthology.orchestra.run.smithi060.stdout:ntpd: time slew -0.001685s 2020-06-04T08:22:20.382 INFO:teuthology.orchestra.run.smithi060.stdout: remote refid st t when poll reach delay offset jitter 2020-06-04T08:22:20.383 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================== 2020-06-04T08:22:20.383 INFO:teuthology.orchestra.run.smithi060.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-06-04T08:22:20.384 INFO:teuthology.orchestra.run.smithi060.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-06-04T08:22:20.384 INFO:teuthology.orchestra.run.smithi060.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-06-04T08:22:20.384 INFO:teuthology.orchestra.run.smithi060.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-06-04T08:22:20.385 INFO:teuthology.orchestra.run:Running command with timeout 60 2020-06-04T08:22:20.388 INFO:teuthology.orchestra.run.smithi196:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-06-04T08:22:20.456 INFO:teuthology.orchestra.run.smithi196.stdout: 4 Jun 08:22:20 ntpd[8724]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-06-04T08:22:20.456 INFO:teuthology.orchestra.run.smithi196.stdout: 4 Jun 08:22:20 ntpd[8724]: Command line: ntpd -gq 2020-06-04T08:22:20.457 INFO:teuthology.orchestra.run.smithi196.stdout: 4 Jun 08:22:20 ntpd[8724]: proto: precision = 0.123 usec (-23) 2020-06-04T08:22:20.457 INFO:teuthology.orchestra.run.smithi196.stdout: 4 Jun 08:22:20 ntpd[8724]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-06-04T08:22:20.457 INFO:teuthology.orchestra.run.smithi196.stdout: 4 Jun 08:22:20 ntpd[8724]: restrict ::: KOD does nothing without LIMITED. 2020-06-04T08:22:20.457 INFO:teuthology.orchestra.run.smithi196.stdout: 4 Jun 08:22:20 ntpd[8724]: Listen and drop on 0 v6wildcard [::]:123 2020-06-04T08:22:20.458 INFO:teuthology.orchestra.run.smithi196.stdout: 4 Jun 08:22:20 ntpd[8724]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-06-04T08:22:20.458 INFO:teuthology.orchestra.run.smithi196.stdout: 4 Jun 08:22:20 ntpd[8724]: Listen normally on 2 lo 127.0.0.1:123 2020-06-04T08:22:20.458 INFO:teuthology.orchestra.run.smithi196.stdout: 4 Jun 08:22:20 ntpd[8724]: Listen normally on 3 enp3s0f1 172.21.15.196:123 2020-06-04T08:22:20.458 INFO:teuthology.orchestra.run.smithi196.stdout: 4 Jun 08:22:20 ntpd[8724]: Listen normally on 4 lo [::1]:123 2020-06-04T08:22:20.459 INFO:teuthology.orchestra.run.smithi196.stdout: 4 Jun 08:22:20 ntpd[8724]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f47%5]:123 2020-06-04T08:22:20.459 INFO:teuthology.orchestra.run.smithi196.stdout: 4 Jun 08:22:20 ntpd[8724]: Listening on routing socket on fd #22 for interface updates 2020-06-04T08:22:20.459 INFO:teuthology.orchestra.run.smithi196.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-06-04T08:22:20.460 INFO:teuthology.orchestra.run.smithi196.stderr:restrict ::: KOD does nothing without LIMITED. 2020-06-04T08:22:21.451 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:21 ntpd[8724]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-04T08:22:21.451 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:21 ntpd[8724]: can't open /var/log/ntpstats/rawstats.20200604: Permission denied 2020-06-04T08:22:21.452 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:21 ntpd[8724]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-04T08:22:21.452 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:21 ntpd[8724]: can't open /var/log/ntpstats/peerstats.20200604: Permission denied 2020-06-04T08:22:21.452 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:21 ntpd[8724]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-04T08:22:21.453 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:21 ntpd[8724]: can't open /var/log/ntpstats/rawstats.20200604: Permission denied 2020-06-04T08:22:21.453 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:21 ntpd[8724]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-04T08:22:21.453 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:21 ntpd[8724]: can't open /var/log/ntpstats/peerstats.20200604: Permission denied 2020-06-04T08:22:22.449 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:22 ntpd[8724]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-04T08:22:22.450 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:22 ntpd[8724]: can't open /var/log/ntpstats/rawstats.20200604: Permission denied 2020-06-04T08:22:22.450 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:22 ntpd[8724]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-04T08:22:22.450 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:22 ntpd[8724]: can't open /var/log/ntpstats/peerstats.20200604: Permission denied 2020-06-04T08:22:23.450 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:23 ntpd[8724]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-04T08:22:23.450 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:23 ntpd[8724]: can't open /var/log/ntpstats/rawstats.20200604: Permission denied 2020-06-04T08:22:23.451 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:23 ntpd[8724]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-04T08:22:23.451 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:23 ntpd[8724]: can't open /var/log/ntpstats/peerstats.20200604: Permission denied 2020-06-04T08:22:23.451 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:23 ntpd[8724]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-04T08:22:23.451 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:23 ntpd[8724]: can't open /var/log/ntpstats/rawstats.20200604: Permission denied 2020-06-04T08:22:23.451 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:23 ntpd[8724]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-04T08:22:23.452 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:23 ntpd[8724]: can't open /var/log/ntpstats/peerstats.20200604: Permission denied 2020-06-04T08:22:24.448 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:24 ntpd[8724]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-04T08:22:24.449 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:24 ntpd[8724]: can't open /var/log/ntpstats/rawstats.20200604: Permission denied 2020-06-04T08:22:24.449 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:24 ntpd[8724]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-04T08:22:24.449 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:24 ntpd[8724]: can't open /var/log/ntpstats/peerstats.20200604: Permission denied 2020-06-04T08:22:25.448 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:25 ntpd[8724]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-04T08:22:25.449 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:25 ntpd[8724]: can't open /var/log/ntpstats/rawstats.20200604: Permission denied 2020-06-04T08:22:25.449 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:25 ntpd[8724]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-04T08:22:25.450 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:25 ntpd[8724]: can't open /var/log/ntpstats/peerstats.20200604: Permission denied 2020-06-04T08:22:25.450 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:25 ntpd[8724]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-04T08:22:25.450 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:25 ntpd[8724]: can't open /var/log/ntpstats/rawstats.20200604: Permission denied 2020-06-04T08:22:25.450 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:25 ntpd[8724]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-04T08:22:25.451 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:25 ntpd[8724]: can't open /var/log/ntpstats/peerstats.20200604: Permission denied 2020-06-04T08:22:25.451 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:25 ntpd[8724]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-04T08:22:25.451 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:25 ntpd[8724]: can't open /var/log/ntpstats/rawstats.20200604: Permission denied 2020-06-04T08:22:25.451 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:25 ntpd[8724]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-04T08:22:25.451 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:25 ntpd[8724]: can't open /var/log/ntpstats/peerstats.20200604: Permission denied 2020-06-04T08:22:26.448 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:26 ntpd[8724]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-04T08:22:26.448 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:26 ntpd[8724]: can't open /var/log/ntpstats/rawstats.20200604: Permission denied 2020-06-04T08:22:26.448 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:26 ntpd[8724]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-04T08:22:26.448 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:26 ntpd[8724]: can't open /var/log/ntpstats/peerstats.20200604: Permission denied 2020-06-04T08:22:27.447 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:27 ntpd[8724]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-06-04T08:22:27.447 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:27 ntpd[8724]: can't open /var/log/ntpstats/rawstats.20200604: Permission denied 2020-06-04T08:22:27.448 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:27 ntpd[8724]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-06-04T08:22:27.448 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:27 ntpd[8724]: can't open /var/log/ntpstats/peerstats.20200604: Permission denied 2020-06-04T08:22:27.448 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:27 ntpd[8724]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-06-04T08:22:27.448 INFO:teuthology.orchestra.run.smithi196.stderr: 4 Jun 08:22:27 ntpd[8724]: can't open /var/log/ntpstats/loopstats.20200604: Permission denied 2020-06-04T08:22:27.449 INFO:teuthology.orchestra.run.smithi196.stdout: 4 Jun 08:22:27 ntpd[8724]: ntpd: time slew -0.007785 s 2020-06-04T08:22:27.449 INFO:teuthology.orchestra.run.smithi196.stdout:ntpd: time slew -0.007785s 2020-06-04T08:22:27.495 INFO:teuthology.orchestra.run.smithi196.stdout: remote refid st t when poll reach delay offset jitter 2020-06-04T08:22:27.495 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================== 2020-06-04T08:22:27.496 INFO:teuthology.orchestra.run.smithi196.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-06-04T08:22:27.496 INFO:teuthology.orchestra.run.smithi196.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-06-04T08:22:27.496 INFO:teuthology.orchestra.run.smithi196.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-06-04T08:22:27.497 INFO:teuthology.orchestra.run.smithi196.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-06-04T08:22:27.498 INFO:teuthology.run_tasks:Running task install... 2020-06-04T08:22:27.513 DEBUG:teuthology.task.install:project ceph 2020-06-04T08:22:27.514 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '5a141c6290e05d95ab4ea2df3c0cb561d6b8e984'}} 2020-06-04T08:22:27.515 DEBUG:teuthology.task.install:config {'sha1': '5a141c6290e05d95ab4ea2df3c0cb561d6b8e984'} 2020-06-04T08:22:27.516 INFO:teuthology.task.install:Using flavor: basic 2020-06-04T08:22:27.528 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-06-04T08:22:27.528 INFO:teuthology.task.install:extra packages: [] 2020-06-04T08:22:27.529 INFO:teuthology.orchestra.run.smithi060:> sudo lsb_release -is 2020-06-04T08:22:27.583 INFO:teuthology.orchestra.run.smithi060.stdout:Ubuntu 2020-06-04T08:22:27.588 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-06-04T08:22:27.589 INFO:teuthology.orchestra.run.smithi196:> sudo lsb_release -is 2020-06-04T08:22:27.591 INFO:teuthology.orchestra.run.smithi060:> sudo apt-key list | grep Ceph 2020-06-04T08:22:27.654 INFO:teuthology.orchestra.run.smithi196.stdout:Ubuntu 2020-06-04T08:22:27.658 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-06-04T08:22:27.661 INFO:teuthology.orchestra.run.smithi196:> sudo apt-key list | grep Ceph 2020-06-04T08:22:27.686 INFO:teuthology.orchestra.run.smithi060.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-06-04T08:22:27.743 INFO:teuthology.orchestra.run.smithi060.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-06-04T08:22:27.744 INFO:teuthology.orchestra.run.smithi060.stdout:uid [ unknown] Ceph.com (release key) 2020-06-04T08:22:27.745 INFO:teuthology.orchestra.run.smithi060:> sudo lsb_release -is 2020-06-04T08:22:27.775 INFO:teuthology.orchestra.run.smithi196.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-06-04T08:22:27.797 INFO:teuthology.orchestra.run.smithi060.stdout:Ubuntu 2020-06-04T08:22:27.813 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-06-04T08:22:27.822 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-06-04T08:22:27.823 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=5a141c6290e05d95ab4ea2df3c0cb561d6b8e984 2020-06-04T08:22:27.842 INFO:teuthology.orchestra.run.smithi196.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-06-04T08:22:27.843 INFO:teuthology.orchestra.run.smithi196.stdout:uid [ unknown] Ceph.com (release key) 2020-06-04T08:22:27.843 INFO:teuthology.orchestra.run.smithi196:> sudo lsb_release -is 2020-06-04T08:22:27.903 INFO:teuthology.orchestra.run.smithi196.stdout:Ubuntu 2020-06-04T08:22:27.908 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-06-04T08:22:27.909 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-06-04T08:22:27.909 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=5a141c6290e05d95ab4ea2df3c0cb561d6b8e984 2020-06-04T08:22:28.091 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default/ 2020-06-04T08:22:28.092 INFO:teuthology.task.install.deb:Package version is 16.0.0-2160-g5a141c6290e-1bionic 2020-06-04T08:22:28.127 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default/ 2020-06-04T08:22:28.128 INFO:teuthology.task.install.deb:Package version is 16.0.0-2160-g5a141c6290e-1bionic 2020-06-04T08:22:28.290 INFO:teuthology.orchestra.run.smithi196:> sudo sh -c 'cat > /etc/apt/sources.list.d/ceph.list' 2020-06-04T08:22:28.293 INFO:teuthology.orchestra.run.smithi060:> sudo sh -c 'cat > /etc/apt/sources.list.d/ceph.list' 2020-06-04T08:22:28.374 INFO:teuthology.orchestra.run.smithi196:> sudo apt-get update 2020-06-04T08:22:28.383 INFO:teuthology.orchestra.run.smithi060:> sudo apt-get update 2020-06-04T08:22:28.587 INFO:teuthology.orchestra.run.smithi196.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-06-04T08:22:28.588 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-06-04T08:22:28.663 INFO:teuthology.orchestra.run.smithi196.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic InRelease 2020-06-04T08:22:28.666 INFO:teuthology.orchestra.run.smithi060.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic InRelease 2020-06-04T08:22:28.679 INFO:teuthology.orchestra.run.smithi196.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-06-04T08:22:28.705 INFO:teuthology.orchestra.run.smithi196.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic Release [7,672 B] 2020-06-04T08:22:28.706 INFO:teuthology.orchestra.run.smithi060.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic Release [7,672 B] 2020-06-04T08:22:28.706 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-06-04T08:22:28.711 INFO:teuthology.orchestra.run.smithi196.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic Release.gpg 2020-06-04T08:22:28.714 INFO:teuthology.orchestra.run.smithi060.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic Release.gpg 2020-06-04T08:22:28.735 INFO:teuthology.orchestra.run.smithi196.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.9 kB] 2020-06-04T08:22:28.738 INFO:teuthology.orchestra.run.smithi060.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,072 B] 2020-06-04T08:22:28.760 INFO:teuthology.orchestra.run.smithi196.stdout:Hit:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-06-04T08:22:28.769 INFO:teuthology.orchestra.run.smithi196.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,072 B] 2020-06-04T08:22:28.773 INFO:teuthology.orchestra.run.smithi060.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.9 kB] 2020-06-04T08:22:28.777 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-06-04T08:22:28.846 INFO:teuthology.orchestra.run.smithi196.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-06-04T08:22:28.869 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-06-04T08:22:29.173 INFO:teuthology.orchestra.run.smithi196.stdout:Fetched 26.6 kB in 1s (45.2 kB/s) 2020-06-04T08:22:29.205 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 26.6 kB in 1s (44.1 kB/s) 2020-06-04T08:22:30.430 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2020-06-04T08:22:30.479 INFO:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-2160-g5a141c6290e-1bionic cephadm=16.0.0-2160-g5a141c6290e-1bionic ceph-mds=16.0.0-2160-g5a141c6290e-1bionic ceph-mgr=16.0.0-2160-g5a141c6290e-1bionic ceph-common=16.0.0-2160-g5a141c6290e-1bionic ceph-fuse=16.0.0-2160-g5a141c6290e-1bionic ceph-test=16.0.0-2160-g5a141c6290e-1bionic radosgw=16.0.0-2160-g5a141c6290e-1bionic python3-rados=16.0.0-2160-g5a141c6290e-1bionic python3-rgw=16.0.0-2160-g5a141c6290e-1bionic python3-cephfs=16.0.0-2160-g5a141c6290e-1bionic python3-rbd=16.0.0-2160-g5a141c6290e-1bionic libcephfs2=16.0.0-2160-g5a141c6290e-1bionic libcephfs-dev=16.0.0-2160-g5a141c6290e-1bionic librados2=16.0.0-2160-g5a141c6290e-1bionic librbd1=16.0.0-2160-g5a141c6290e-1bionic rbd-fuse=16.0.0-2160-g5a141c6290e-1bionic 2020-06-04T08:22:30.514 INFO:teuthology.orchestra.run.smithi196.stdout:Reading package lists... 2020-06-04T08:22:30.558 INFO:teuthology.orchestra.run.smithi196:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.0.0-2160-g5a141c6290e-1bionic cephadm=16.0.0-2160-g5a141c6290e-1bionic ceph-mds=16.0.0-2160-g5a141c6290e-1bionic ceph-mgr=16.0.0-2160-g5a141c6290e-1bionic ceph-common=16.0.0-2160-g5a141c6290e-1bionic ceph-fuse=16.0.0-2160-g5a141c6290e-1bionic ceph-test=16.0.0-2160-g5a141c6290e-1bionic radosgw=16.0.0-2160-g5a141c6290e-1bionic python3-rados=16.0.0-2160-g5a141c6290e-1bionic python3-rgw=16.0.0-2160-g5a141c6290e-1bionic python3-cephfs=16.0.0-2160-g5a141c6290e-1bionic python3-rbd=16.0.0-2160-g5a141c6290e-1bionic libcephfs2=16.0.0-2160-g5a141c6290e-1bionic libcephfs-dev=16.0.0-2160-g5a141c6290e-1bionic librados2=16.0.0-2160-g5a141c6290e-1bionic librbd1=16.0.0-2160-g5a141c6290e-1bionic rbd-fuse=16.0.0-2160-g5a141c6290e-1bionic 2020-06-04T08:22:30.563 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2020-06-04T08:22:30.640 INFO:teuthology.orchestra.run.smithi196.stdout:Reading package lists... 2020-06-04T08:22:30.820 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2020-06-04T08:22:30.821 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2020-06-04T08:22:30.870 INFO:teuthology.orchestra.run.smithi196.stdout:Building dependency tree... 2020-06-04T08:22:30.871 INFO:teuthology.orchestra.run.smithi196.stdout:Reading state information... 2020-06-04T08:22:31.014 INFO:teuthology.orchestra.run.smithi060.stdout:The following additional packages will be installed: 2020-06-04T08:22:31.014 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-06-04T08:22:31.014 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-06-04T08:22:31.014 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook ceph-mon ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 2020-06-04T08:22:31.015 INFO:teuthology.orchestra.run.smithi060.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2020-06-04T08:22:31.015 INFO:teuthology.orchestra.run.smithi060.stdout: libsnappy1v5 libxslt1.1 python-attr python-funcsigs python-pastedeploy-tpl 2020-06-04T08:22:31.015 INFO:teuthology.orchestra.run.smithi060.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-06-04T08:22:31.015 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-06-04T08:22:31.015 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-06-04T08:22:31.016 INFO:teuthology.orchestra.run.smithi060.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-06-04T08:22:31.016 INFO:teuthology.orchestra.run.smithi060.stdout: python3-httplib2 python3-joblib python3-json-pointer python3-jsonpatch 2020-06-04T08:22:31.016 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-06-04T08:22:31.023 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-06-04T08:22:31.024 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-06-04T08:22:31.024 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-06-04T08:22:31.024 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-06-04T08:22:31.024 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-06-04T08:22:31.024 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-06-04T08:22:31.025 INFO:teuthology.orchestra.run.smithi060.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-06-04T08:22:31.025 INFO:teuthology.orchestra.run.smithi060.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-06-04T08:22:31.025 INFO:teuthology.orchestra.run.smithi060.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-06-04T08:22:31.025 INFO:teuthology.orchestra.run.smithi060.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-06-04T08:22:31.025 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-06-04T08:22:31.026 INFO:teuthology.orchestra.run.smithi060.stdout:Suggested packages: 2020-06-04T08:22:31.026 INFO:teuthology.orchestra.run.smithi060.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-06-04T08:22:31.026 INFO:teuthology.orchestra.run.smithi060.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-06-04T08:22:31.026 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi python3-crypto 2020-06-04T08:22:31.026 INFO:teuthology.orchestra.run.smithi060.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc dvipng 2020-06-04T08:22:31.027 INFO:teuthology.orchestra.run.smithi060.stdout: ffmpeg gir1.2-gtk-3.0 ghostscript inkscape ipython3 python-matplotlib-doc 2020-06-04T08:22:31.027 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cairocffi python3-gi-cairo python3-gobject python3-pyqt4 python3-sip 2020-06-04T08:22:31.027 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tornado texlive-extra-utils texlive-latex-extra ttf-staypuft 2020-06-04T08:22:31.027 INFO:teuthology.orchestra.run.smithi060.stdout: python-numpy-doc python3-dev python3-numpy-dbg python-openssl-doc 2020-06-04T08:22:31.027 INFO:teuthology.orchestra.run.smithi060.stdout: python3-openssl-dbg httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2020-06-04T08:22:31.028 INFO:teuthology.orchestra.run.smithi060.stdout: libjs-mochikit python-pecan-doc python-pil-doc python3-pil-dbg 2020-06-04T08:22:31.028 INFO:teuthology.orchestra.run.smithi060.stdout: python-pyinotify-doc python-pyparsing-doc python-scipy-doc 2020-06-04T08:22:31.028 INFO:teuthology.orchestra.run.smithi060.stdout: python-setuptools-doc python3-dap python-sklearn-doc python-sqlalchemy-doc 2020-06-04T08:22:31.028 INFO:teuthology.orchestra.run.smithi060.stdout: python3-psycopg2 python3-pymysql python3-fdb tix python3-tk-dbg 2020-06-04T08:22:31.028 INFO:teuthology.orchestra.run.smithi060.stdout: python-waitress-doc python-webob-doc python-webtest-doc python3-termcolor 2020-06-04T08:22:31.029 INFO:teuthology.orchestra.run.smithi060.stdout: python3-watchdog python-werkzeug-doc 2020-06-04T08:22:31.031 INFO:teuthology.orchestra.run.smithi196.stdout:The following additional packages will be installed: 2020-06-04T08:22:31.032 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-06-04T08:22:31.032 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-06-04T08:22:31.032 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook ceph-mon ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 2020-06-04T08:22:31.032 INFO:teuthology.orchestra.run.smithi196.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2020-06-04T08:22:31.032 INFO:teuthology.orchestra.run.smithi196.stdout: libsnappy1v5 libxslt1.1 python-attr python-funcsigs python-pastedeploy-tpl 2020-06-04T08:22:31.033 INFO:teuthology.orchestra.run.smithi196.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-06-04T08:22:31.033 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-06-04T08:22:31.033 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-06-04T08:22:31.034 INFO:teuthology.orchestra.run.smithi196.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-06-04T08:22:31.034 INFO:teuthology.orchestra.run.smithi196.stdout: python3-httplib2 python3-joblib python3-json-pointer python3-jsonpatch 2020-06-04T08:22:31.034 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-06-04T08:22:31.034 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-06-04T08:22:31.034 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-06-04T08:22:31.035 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-06-04T08:22:31.035 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-06-04T08:22:31.035 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-06-04T08:22:31.035 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-06-04T08:22:31.035 INFO:teuthology.orchestra.run.smithi196.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-06-04T08:22:31.036 INFO:teuthology.orchestra.run.smithi196.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-06-04T08:22:31.036 INFO:teuthology.orchestra.run.smithi196.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-06-04T08:22:31.036 INFO:teuthology.orchestra.run.smithi196.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-06-04T08:22:31.036 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-06-04T08:22:31.036 INFO:teuthology.orchestra.run.smithi196.stdout:Suggested packages: 2020-06-04T08:22:31.037 INFO:teuthology.orchestra.run.smithi196.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-06-04T08:22:31.037 INFO:teuthology.orchestra.run.smithi196.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-06-04T08:22:31.037 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi python3-crypto 2020-06-04T08:22:31.037 INFO:teuthology.orchestra.run.smithi196.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc dvipng 2020-06-04T08:22:31.037 INFO:teuthology.orchestra.run.smithi196.stdout: ffmpeg gir1.2-gtk-3.0 ghostscript inkscape ipython3 python-matplotlib-doc 2020-06-04T08:22:31.037 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cairocffi python3-gi-cairo python3-gobject python3-pyqt4 python3-sip 2020-06-04T08:22:31.038 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tornado texlive-extra-utils texlive-latex-extra ttf-staypuft 2020-06-04T08:22:31.038 INFO:teuthology.orchestra.run.smithi196.stdout: python-numpy-doc python3-dev python3-numpy-dbg python-openssl-doc 2020-06-04T08:22:31.038 INFO:teuthology.orchestra.run.smithi196.stdout: python3-openssl-dbg httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2020-06-04T08:22:31.038 INFO:teuthology.orchestra.run.smithi196.stdout: libjs-mochikit python-pecan-doc python-pil-doc python3-pil-dbg 2020-06-04T08:22:31.038 INFO:teuthology.orchestra.run.smithi196.stdout: python-pyinotify-doc python-pyparsing-doc python-scipy-doc 2020-06-04T08:22:31.039 INFO:teuthology.orchestra.run.smithi196.stdout: python-setuptools-doc python3-dap python-sklearn-doc python-sqlalchemy-doc 2020-06-04T08:22:31.039 INFO:teuthology.orchestra.run.smithi196.stdout: python3-psycopg2 python3-pymysql python3-fdb tix python3-tk-dbg 2020-06-04T08:22:31.039 INFO:teuthology.orchestra.run.smithi196.stdout: python-waitress-doc python-webob-doc python-webtest-doc python3-termcolor 2020-06-04T08:22:31.040 INFO:teuthology.orchestra.run.smithi196.stdout: python3-watchdog python-werkzeug-doc 2020-06-04T08:22:31.107 INFO:teuthology.orchestra.run.smithi060.stdout:The following NEW packages will be installed: 2020-06-04T08:22:31.108 INFO:teuthology.orchestra.run.smithi060.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-06-04T08:22:31.108 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-06-04T08:22:31.108 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-06-04T08:22:31.109 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2020-06-04T08:22:31.110 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-06-04T08:22:31.111 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr 2020-06-04T08:22:31.111 INFO:teuthology.orchestra.run.smithi060.stdout: python-funcsigs python-pastedeploy-tpl python-pluggy python-py python-pytest 2020-06-04T08:22:31.111 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-06-04T08:22:31.111 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2020-06-04T08:22:31.112 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-06-04T08:22:31.112 INFO:teuthology.orchestra.run.smithi060.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-06-04T08:22:31.112 INFO:teuthology.orchestra.run.smithi060.stdout: python3-httplib2 python3-joblib python3-json-pointer python3-jsonpatch 2020-06-04T08:22:31.112 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-06-04T08:22:31.112 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-06-04T08:22:31.113 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-06-04T08:22:31.113 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-06-04T08:22:31.113 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-06-04T08:22:31.113 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-06-04T08:22:31.114 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados python3-rbd python3-repoze.lru python3-rgw python3-routes 2020-06-04T08:22:31.114 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa python3-scipy python3-setuptools python3-simplegeneric 2020-06-04T08:22:31.114 INFO:teuthology.orchestra.run.smithi060.stdout: python3-simplejson python3-singledispatch python3-sklearn 2020-06-04T08:22:31.114 INFO:teuthology.orchestra.run.smithi060.stdout: python3-sklearn-lib python3-sqlalchemy python3-tempita python3-tk python3-tz 2020-06-04T08:22:31.115 INFO:teuthology.orchestra.run.smithi060.stdout: python3-uritemplate python3-waitress python3-webencodings python3-webob 2020-06-04T08:22:31.115 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket python3-webtest python3-werkzeug radosgw rbd-fuse socat 2020-06-04T08:22:31.115 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet 2020-06-04T08:22:31.115 INFO:teuthology.orchestra.run.smithi060.stdout:The following packages will be upgraded: 2020-06-04T08:22:31.115 INFO:teuthology.orchestra.run.smithi060.stdout: librados2 librbd1 2020-06-04T08:22:31.127 INFO:teuthology.orchestra.run.smithi196.stdout:The following NEW packages will be installed: 2020-06-04T08:22:31.128 INFO:teuthology.orchestra.run.smithi196.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-06-04T08:22:31.128 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-06-04T08:22:31.128 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-06-04T08:22:31.128 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2020-06-04T08:22:31.130 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-06-04T08:22:31.130 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr 2020-06-04T08:22:31.130 INFO:teuthology.orchestra.run.smithi196.stdout: python-funcsigs python-pastedeploy-tpl python-pluggy python-py python-pytest 2020-06-04T08:22:31.131 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-06-04T08:22:31.131 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2020-06-04T08:22:31.131 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-06-04T08:22:31.131 INFO:teuthology.orchestra.run.smithi196.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-06-04T08:22:31.131 INFO:teuthology.orchestra.run.smithi196.stdout: python3-httplib2 python3-joblib python3-json-pointer python3-jsonpatch 2020-06-04T08:22:31.132 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2020-06-04T08:22:31.132 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2020-06-04T08:22:31.132 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2020-06-04T08:22:31.132 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2020-06-04T08:22:31.132 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2020-06-04T08:22:31.132 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2020-06-04T08:22:31.133 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados python3-rbd python3-repoze.lru python3-rgw python3-routes 2020-06-04T08:22:31.133 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa python3-scipy python3-setuptools python3-simplegeneric 2020-06-04T08:22:31.133 INFO:teuthology.orchestra.run.smithi196.stdout: python3-simplejson python3-singledispatch python3-sklearn 2020-06-04T08:22:31.133 INFO:teuthology.orchestra.run.smithi196.stdout: python3-sklearn-lib python3-sqlalchemy python3-tempita python3-tk python3-tz 2020-06-04T08:22:31.133 INFO:teuthology.orchestra.run.smithi196.stdout: python3-uritemplate python3-waitress python3-webencodings python3-webob 2020-06-04T08:22:31.134 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket python3-webtest python3-werkzeug radosgw rbd-fuse socat 2020-06-04T08:22:31.134 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet 2020-06-04T08:22:31.134 INFO:teuthology.orchestra.run.smithi196.stdout:The following packages will be upgraded: 2020-06-04T08:22:31.134 INFO:teuthology.orchestra.run.smithi196.stdout: librados2 librbd1 2020-06-04T08:22:31.273 INFO:teuthology.orchestra.run.smithi060.stdout:2 upgraded, 109 newly installed, 0 to remove and 47 not upgraded. 2020-06-04T08:22:31.274 INFO:teuthology.orchestra.run.smithi060.stdout:Need to get 163 MB of archives. 2020-06-04T08:22:31.274 INFO:teuthology.orchestra.run.smithi060.stdout:After this operation, 761 MB of additional disk space will be used. 2020-06-04T08:22:31.274 INFO:teuthology.orchestra.run.smithi060.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-2160-g5a141c6290e-1bionic [1,675 kB] 2020-06-04T08:22:31.294 INFO:teuthology.orchestra.run.smithi196.stdout:2 upgraded, 109 newly installed, 0 to remove and 47 not upgraded. 2020-06-04T08:22:31.294 INFO:teuthology.orchestra.run.smithi196.stdout:Need to get 163 MB of archives. 2020-06-04T08:22:31.295 INFO:teuthology.orchestra.run.smithi196.stdout:After this operation, 761 MB of additional disk space will be used. 2020-06-04T08:22:31.295 INFO:teuthology.orchestra.run.smithi196.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.0.0-2160-g5a141c6290e-1bionic [1,675 kB] 2020-06-04T08:22:31.301 INFO:teuthology.orchestra.run.smithi060.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-06-04T08:22:31.321 INFO:teuthology.orchestra.run.smithi196.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-06-04T08:22:31.383 INFO:teuthology.orchestra.run.smithi060.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-06-04T08:22:31.403 INFO:teuthology.orchestra.run.smithi196.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-06-04T08:22:31.467 INFO:teuthology.orchestra.run.smithi060.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-2160-g5a141c6290e-1bionic [3,155 kB] 2020-06-04T08:22:31.483 INFO:teuthology.orchestra.run.smithi196.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.0.0-2160-g5a141c6290e-1bionic [3,155 kB] 2020-06-04T08:22:31.645 INFO:teuthology.orchestra.run.smithi060.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-06-04T08:22:31.651 INFO:teuthology.orchestra.run.smithi060.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-2160-g5a141c6290e-1bionic [463 kB] 2020-06-04T08:22:31.652 INFO:teuthology.orchestra.run.smithi060.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-06-04T08:22:31.659 INFO:teuthology.orchestra.run.smithi060.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-06-04T08:22:31.661 INFO:teuthology.orchestra.run.smithi196.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-06-04T08:22:31.668 INFO:teuthology.orchestra.run.smithi196.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.0.0-2160-g5a141c6290e-1bionic [463 kB] 2020-06-04T08:22:31.671 INFO:teuthology.orchestra.run.smithi196.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-06-04T08:22:31.682 INFO:teuthology.orchestra.run.smithi196.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-06-04T08:22:31.683 INFO:teuthology.orchestra.run.smithi060.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-2160-g5a141c6290e-1bionic [346 kB] 2020-06-04T08:22:31.702 INFO:teuthology.orchestra.run.smithi196.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.0.0-2160-g5a141c6290e-1bionic [346 kB] 2020-06-04T08:22:31.713 INFO:teuthology.orchestra.run.smithi060.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-2160-g5a141c6290e-1bionic [20.6 kB] 2020-06-04T08:22:31.713 INFO:teuthology.orchestra.run.smithi060.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-2160-g5a141c6290e-1bionic [132 kB] 2020-06-04T08:22:31.718 INFO:teuthology.orchestra.run.smithi060.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-2160-g5a141c6290e-1bionic [24.2 kB] 2020-06-04T08:22:31.719 INFO:teuthology.orchestra.run.smithi060.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-2160-g5a141c6290e-1bionic [309 kB] 2020-06-04T08:22:31.733 INFO:teuthology.orchestra.run.smithi196.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.0.0-2160-g5a141c6290e-1bionic [20.6 kB] 2020-06-04T08:22:31.734 INFO:teuthology.orchestra.run.smithi196.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.0.0-2160-g5a141c6290e-1bionic [132 kB] 2020-06-04T08:22:31.738 INFO:teuthology.orchestra.run.smithi196.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.0.0-2160-g5a141c6290e-1bionic [24.2 kB] 2020-06-04T08:22:31.739 INFO:teuthology.orchestra.run.smithi196.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.0.0-2160-g5a141c6290e-1bionic [309 kB] 2020-06-04T08:22:31.758 INFO:teuthology.orchestra.run.smithi060.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-2160-g5a141c6290e-1bionic [2,927 kB] 2020-06-04T08:22:31.773 INFO:teuthology.orchestra.run.smithi196.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.0.0-2160-g5a141c6290e-1bionic [2,927 kB] 2020-06-04T08:22:31.773 INFO:teuthology.orchestra.run.smithi060.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-06-04T08:22:31.773 INFO:teuthology.orchestra.run.smithi060.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-06-04T08:22:31.784 INFO:teuthology.orchestra.run.smithi196.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-06-04T08:22:31.787 INFO:teuthology.orchestra.run.smithi196.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-06-04T08:22:31.801 INFO:teuthology.orchestra.run.smithi060.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-06-04T08:22:31.811 INFO:teuthology.orchestra.run.smithi060.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-06-04T08:22:31.819 INFO:teuthology.orchestra.run.smithi060.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-06-04T08:22:31.831 INFO:teuthology.orchestra.run.smithi060.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-06-04T08:22:31.833 INFO:teuthology.orchestra.run.smithi196.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-06-04T08:22:31.840 INFO:teuthology.orchestra.run.smithi060.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.3 [221 kB] 2020-06-04T08:22:31.841 INFO:teuthology.orchestra.run.smithi196.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-06-04T08:22:31.847 INFO:teuthology.orchestra.run.smithi196.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-06-04T08:22:31.857 INFO:teuthology.orchestra.run.smithi196.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-06-04T08:22:31.866 INFO:teuthology.orchestra.run.smithi196.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.3 [221 kB] 2020-06-04T08:22:31.867 INFO:teuthology.orchestra.run.smithi060.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-06-04T08:22:31.869 INFO:teuthology.orchestra.run.smithi060.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-06-04T08:22:31.889 INFO:teuthology.orchestra.run.smithi196.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-06-04T08:22:31.889 INFO:teuthology.orchestra.run.smithi060.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-06-04T08:22:31.892 INFO:teuthology.orchestra.run.smithi196.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-06-04T08:22:31.900 INFO:teuthology.orchestra.run.smithi060.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-06-04T08:22:31.902 INFO:teuthology.orchestra.run.smithi060.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-06-04T08:22:31.905 INFO:teuthology.orchestra.run.smithi196.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-06-04T08:22:31.919 INFO:teuthology.orchestra.run.smithi196.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-06-04T08:22:31.932 INFO:teuthology.orchestra.run.smithi060.stdout:Get:27 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-2160-g5a141c6290e-1bionic [99.1 kB] 2020-06-04T08:22:31.934 INFO:teuthology.orchestra.run.smithi060.stdout:Get:28 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-2160-g5a141c6290e-1bionic [418 kB] 2020-06-04T08:22:31.957 INFO:teuthology.orchestra.run.smithi196.stdout:Get:26 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.0.0-2160-g5a141c6290e-1bionic [99.1 kB] 2020-06-04T08:22:31.960 INFO:teuthology.orchestra.run.smithi196.stdout:Get:27 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.0.0-2160-g5a141c6290e-1bionic [418 kB] 2020-06-04T08:22:31.969 INFO:teuthology.orchestra.run.smithi060.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-06-04T08:22:31.970 INFO:teuthology.orchestra.run.smithi060.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-06-04T08:22:31.973 INFO:teuthology.orchestra.run.smithi060.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-06-04T08:22:31.973 INFO:teuthology.orchestra.run.smithi060.stdout:Get:32 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-2160-g5a141c6290e-1bionic [19.5 MB] 2020-06-04T08:22:31.974 INFO:teuthology.orchestra.run.smithi060.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-06-04T08:22:31.974 INFO:teuthology.orchestra.run.smithi060.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-06-04T08:22:31.977 INFO:teuthology.orchestra.run.smithi060.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-06-04T08:22:31.979 INFO:teuthology.orchestra.run.smithi060.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-06-04T08:22:31.993 INFO:teuthology.orchestra.run.smithi196.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-06-04T08:22:31.994 INFO:teuthology.orchestra.run.smithi196.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-06-04T08:22:31.994 INFO:teuthology.orchestra.run.smithi196.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-06-04T08:22:31.994 INFO:teuthology.orchestra.run.smithi196.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-06-04T08:22:31.994 INFO:teuthology.orchestra.run.smithi196.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-06-04T08:22:31.994 INFO:teuthology.orchestra.run.smithi196.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-06-04T08:22:31.995 INFO:teuthology.orchestra.run.smithi196.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-06-04T08:22:31.996 INFO:teuthology.orchestra.run.smithi196.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-06-04T08:22:31.999 INFO:teuthology.orchestra.run.smithi196.stdout:Get:36 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.0.0-2160-g5a141c6290e-1bionic [19.5 MB] 2020-06-04T08:22:32.054 INFO:teuthology.orchestra.run.smithi060.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-06-04T08:22:32.055 INFO:teuthology.orchestra.run.smithi060.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-06-04T08:22:32.064 INFO:teuthology.orchestra.run.smithi060.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-06-04T08:22:32.068 INFO:teuthology.orchestra.run.smithi196.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-06-04T08:22:32.069 INFO:teuthology.orchestra.run.smithi196.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-06-04T08:22:32.131 INFO:teuthology.orchestra.run.smithi060.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-06-04T08:22:32.132 INFO:teuthology.orchestra.run.smithi060.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-06-04T08:22:32.135 INFO:teuthology.orchestra.run.smithi060.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-06-04T08:22:32.137 INFO:teuthology.orchestra.run.smithi060.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-06-04T08:22:32.142 INFO:teuthology.orchestra.run.smithi060.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-06-04T08:22:32.142 INFO:teuthology.orchestra.run.smithi060.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-06-04T08:22:32.143 INFO:teuthology.orchestra.run.smithi060.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-06-04T08:22:32.147 INFO:teuthology.orchestra.run.smithi196.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-06-04T08:22:32.148 INFO:teuthology.orchestra.run.smithi196.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-06-04T08:22:32.151 INFO:teuthology.orchestra.run.smithi196.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-06-04T08:22:32.152 INFO:teuthology.orchestra.run.smithi196.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-06-04T08:22:32.153 INFO:teuthology.orchestra.run.smithi196.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-06-04T08:22:32.159 INFO:teuthology.orchestra.run.smithi196.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-06-04T08:22:32.162 INFO:teuthology.orchestra.run.smithi196.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-06-04T08:22:32.162 INFO:teuthology.orchestra.run.smithi196.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-06-04T08:22:32.236 INFO:teuthology.orchestra.run.smithi060.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-06-04T08:22:32.236 INFO:teuthology.orchestra.run.smithi060.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-06-04T08:22:32.242 INFO:teuthology.orchestra.run.smithi196.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-06-04T08:22:32.242 INFO:teuthology.orchestra.run.smithi196.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-06-04T08:22:32.504 INFO:teuthology.orchestra.run.smithi060.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-06-04T08:22:32.520 INFO:teuthology.orchestra.run.smithi196.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-06-04T08:22:32.557 INFO:teuthology.orchestra.run.smithi196.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-06-04T08:22:32.559 INFO:teuthology.orchestra.run.smithi196.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-06-04T08:22:32.575 INFO:teuthology.orchestra.run.smithi060.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-06-04T08:22:32.575 INFO:teuthology.orchestra.run.smithi060.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-06-04T08:22:32.591 INFO:teuthology.orchestra.run.smithi060.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-06-04T08:22:32.591 INFO:teuthology.orchestra.run.smithi060.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-06-04T08:22:32.592 INFO:teuthology.orchestra.run.smithi060.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.1 [28.6 kB] 2020-06-04T08:22:32.592 INFO:teuthology.orchestra.run.smithi060.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-06-04T08:22:32.592 INFO:teuthology.orchestra.run.smithi060.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-06-04T08:22:32.592 INFO:teuthology.orchestra.run.smithi060.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-06-04T08:22:32.592 INFO:teuthology.orchestra.run.smithi060.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-06-04T08:22:32.603 INFO:teuthology.orchestra.run.smithi196.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-06-04T08:22:32.603 INFO:teuthology.orchestra.run.smithi196.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-06-04T08:22:32.604 INFO:teuthology.orchestra.run.smithi196.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.1 [28.6 kB] 2020-06-04T08:22:32.604 INFO:teuthology.orchestra.run.smithi196.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-06-04T08:22:32.605 INFO:teuthology.orchestra.run.smithi196.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-06-04T08:22:32.606 INFO:teuthology.orchestra.run.smithi196.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-06-04T08:22:32.607 INFO:teuthology.orchestra.run.smithi196.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-06-04T08:22:32.670 INFO:teuthology.orchestra.run.smithi060.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-06-04T08:22:32.670 INFO:teuthology.orchestra.run.smithi060.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-06-04T08:22:32.684 INFO:teuthology.orchestra.run.smithi196.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-06-04T08:22:32.685 INFO:teuthology.orchestra.run.smithi196.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-06-04T08:22:32.749 INFO:teuthology.orchestra.run.smithi060.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-06-04T08:22:32.749 INFO:teuthology.orchestra.run.smithi060.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-06-04T08:22:32.754 INFO:teuthology.orchestra.run.smithi060.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-06-04T08:22:32.755 INFO:teuthology.orchestra.run.smithi060.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-06-04T08:22:32.755 INFO:teuthology.orchestra.run.smithi060.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-06-04T08:22:32.759 INFO:teuthology.orchestra.run.smithi060.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-06-04T08:22:32.760 INFO:teuthology.orchestra.run.smithi060.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-06-04T08:22:32.761 INFO:teuthology.orchestra.run.smithi060.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-06-04T08:22:32.763 INFO:teuthology.orchestra.run.smithi196.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-06-04T08:22:32.766 INFO:teuthology.orchestra.run.smithi196.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-06-04T08:22:32.770 INFO:teuthology.orchestra.run.smithi196.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-06-04T08:22:32.771 INFO:teuthology.orchestra.run.smithi196.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-06-04T08:22:32.772 INFO:teuthology.orchestra.run.smithi196.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-06-04T08:22:32.782 INFO:teuthology.orchestra.run.smithi196.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-06-04T08:22:32.783 INFO:teuthology.orchestra.run.smithi196.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-06-04T08:22:32.783 INFO:teuthology.orchestra.run.smithi196.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-06-04T08:22:32.828 INFO:teuthology.orchestra.run.smithi060.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-06-04T08:22:32.832 INFO:teuthology.orchestra.run.smithi060.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-06-04T08:22:32.845 INFO:teuthology.orchestra.run.smithi196.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-06-04T08:22:32.849 INFO:teuthology.orchestra.run.smithi196.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-06-04T08:22:32.911 INFO:teuthology.orchestra.run.smithi060.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-06-04T08:22:32.912 INFO:teuthology.orchestra.run.smithi060.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-06-04T08:22:32.912 INFO:teuthology.orchestra.run.smithi060.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-06-04T08:22:32.912 INFO:teuthology.orchestra.run.smithi060.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-06-04T08:22:32.914 INFO:teuthology.orchestra.run.smithi060.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-06-04T08:22:32.916 INFO:teuthology.orchestra.run.smithi060.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-06-04T08:22:32.916 INFO:teuthology.orchestra.run.smithi060.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-06-04T08:22:32.917 INFO:teuthology.orchestra.run.smithi060.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-06-04T08:22:32.925 INFO:teuthology.orchestra.run.smithi196.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-06-04T08:22:32.926 INFO:teuthology.orchestra.run.smithi196.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-06-04T08:22:32.931 INFO:teuthology.orchestra.run.smithi196.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-06-04T08:22:32.931 INFO:teuthology.orchestra.run.smithi196.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-06-04T08:22:32.931 INFO:teuthology.orchestra.run.smithi196.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-06-04T08:22:32.933 INFO:teuthology.orchestra.run.smithi196.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-06-04T08:22:32.934 INFO:teuthology.orchestra.run.smithi196.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-06-04T08:22:32.935 INFO:teuthology.orchestra.run.smithi196.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-06-04T08:22:32.989 INFO:teuthology.orchestra.run.smithi060.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-06-04T08:22:32.991 INFO:teuthology.orchestra.run.smithi060.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-06-04T08:22:33.008 INFO:teuthology.orchestra.run.smithi196.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-06-04T08:22:33.009 INFO:teuthology.orchestra.run.smithi196.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-06-04T08:22:33.070 INFO:teuthology.orchestra.run.smithi060.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-06-04T08:22:33.071 INFO:teuthology.orchestra.run.smithi060.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-06-04T08:22:33.072 INFO:teuthology.orchestra.run.smithi060.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-06-04T08:22:33.086 INFO:teuthology.orchestra.run.smithi196.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-06-04T08:22:33.089 INFO:teuthology.orchestra.run.smithi196.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-06-04T08:22:33.089 INFO:teuthology.orchestra.run.smithi196.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-06-04T08:22:33.170 INFO:teuthology.orchestra.run.smithi060.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-06-04T08:22:33.171 INFO:teuthology.orchestra.run.smithi060.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-06-04T08:22:33.171 INFO:teuthology.orchestra.run.smithi060.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.2 [329 kB] 2020-06-04T08:22:33.178 INFO:teuthology.orchestra.run.smithi060.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-06-04T08:22:33.179 INFO:teuthology.orchestra.run.smithi060.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-06-04T08:22:33.179 INFO:teuthology.orchestra.run.smithi060.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-06-04T08:22:33.179 INFO:teuthology.orchestra.run.smithi060.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-06-04T08:22:33.187 INFO:teuthology.orchestra.run.smithi196.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-06-04T08:22:33.188 INFO:teuthology.orchestra.run.smithi196.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-06-04T08:22:33.188 INFO:teuthology.orchestra.run.smithi196.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.2 [329 kB] 2020-06-04T08:22:33.192 INFO:teuthology.orchestra.run.smithi196.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-06-04T08:22:33.192 INFO:teuthology.orchestra.run.smithi196.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-06-04T08:22:33.192 INFO:teuthology.orchestra.run.smithi196.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-06-04T08:22:33.193 INFO:teuthology.orchestra.run.smithi196.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-06-04T08:22:33.234 INFO:teuthology.orchestra.run.smithi060.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-2160-g5a141c6290e-1bionic [5,602 kB] 2020-06-04T08:22:33.235 INFO:teuthology.orchestra.run.smithi060.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-06-04T08:22:33.235 INFO:teuthology.orchestra.run.smithi060.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-06-04T08:22:33.248 INFO:teuthology.orchestra.run.smithi196.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-06-04T08:22:33.249 INFO:teuthology.orchestra.run.smithi196.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-06-04T08:22:33.307 INFO:teuthology.orchestra.run.smithi196.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.0.0-2160-g5a141c6290e-1bionic [5,602 kB] 2020-06-04T08:22:33.601 INFO:teuthology.orchestra.run.smithi060.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-2160-g5a141c6290e-1bionic [142 kB] 2020-06-04T08:22:33.610 INFO:teuthology.orchestra.run.smithi060.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-2160-g5a141c6290e-1bionic [1,357 kB] 2020-06-04T08:22:33.653 INFO:teuthology.orchestra.run.smithi196.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.0.0-2160-g5a141c6290e-1bionic [142 kB] 2020-06-04T08:22:33.657 INFO:teuthology.orchestra.run.smithi196.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.0.0-2160-g5a141c6290e-1bionic [1,357 kB] 2020-06-04T08:22:33.681 INFO:teuthology.orchestra.run.smithi060.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-2160-g5a141c6290e-1bionic [6,573 kB] 2020-06-04T08:22:33.770 INFO:teuthology.orchestra.run.smithi196.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.0.0-2160-g5a141c6290e-1bionic [6,573 kB] 2020-06-04T08:22:34.114 INFO:teuthology.orchestra.run.smithi060.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-2160-g5a141c6290e-1bionic [24.6 MB] 2020-06-04T08:22:34.218 INFO:teuthology.orchestra.run.smithi196.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.0.0-2160-g5a141c6290e-1bionic [24.6 MB] 2020-06-04T08:22:35.741 INFO:teuthology.orchestra.run.smithi060.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-2160-g5a141c6290e-1bionic [3,816 B] 2020-06-04T08:22:35.741 INFO:teuthology.orchestra.run.smithi060.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-2160-g5a141c6290e-1bionic [544 kB] 2020-06-04T08:22:35.754 INFO:teuthology.orchestra.run.smithi060.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-2160-g5a141c6290e-1bionic [1,942 kB] 2020-06-04T08:22:35.859 INFO:teuthology.orchestra.run.smithi196.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.0.0-2160-g5a141c6290e-1bionic [3,816 B] 2020-06-04T08:22:35.860 INFO:teuthology.orchestra.run.smithi196.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.0.0-2160-g5a141c6290e-1bionic [544 kB] 2020-06-04T08:22:35.874 INFO:teuthology.orchestra.run.smithi196.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.0.0-2160-g5a141c6290e-1bionic [1,942 kB] 2020-06-04T08:22:35.896 INFO:teuthology.orchestra.run.smithi060.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-2160-g5a141c6290e-1bionic [41.6 kB] 2020-06-04T08:22:35.898 INFO:teuthology.orchestra.run.smithi060.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-2160-g5a141c6290e-1bionic [39.7 kB] 2020-06-04T08:22:35.899 INFO:teuthology.orchestra.run.smithi060.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-2160-g5a141c6290e-1bionic [3,120 kB] 2020-06-04T08:22:36.018 INFO:teuthology.orchestra.run.smithi196.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.0.0-2160-g5a141c6290e-1bionic [41.6 kB] 2020-06-04T08:22:36.019 INFO:teuthology.orchestra.run.smithi196.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.0.0-2160-g5a141c6290e-1bionic [39.7 kB] 2020-06-04T08:22:36.020 INFO:teuthology.orchestra.run.smithi196.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.0.0-2160-g5a141c6290e-1bionic [3,120 kB] 2020-06-04T08:22:36.080 INFO:teuthology.orchestra.run.smithi060.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 16.0.0-2160-g5a141c6290e-1bionic [34.2 kB] 2020-06-04T08:22:36.081 INFO:teuthology.orchestra.run.smithi060.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-2160-g5a141c6290e-1bionic [7,881 kB] 2020-06-04T08:22:36.245 INFO:teuthology.orchestra.run.smithi196.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 16.0.0-2160-g5a141c6290e-1bionic [34.2 kB] 2020-06-04T08:22:36.246 INFO:teuthology.orchestra.run.smithi196.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.0.0-2160-g5a141c6290e-1bionic [7,881 kB] 2020-06-04T08:22:36.621 INFO:teuthology.orchestra.run.smithi060.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-2160-g5a141c6290e-1bionic [3,932 B] 2020-06-04T08:22:36.622 INFO:teuthology.orchestra.run.smithi060.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-2160-g5a141c6290e-1bionic [22.4 kB] 2020-06-04T08:22:36.622 INFO:teuthology.orchestra.run.smithi060.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-2160-g5a141c6290e-1bionic [47.0 MB] 2020-06-04T08:22:36.842 INFO:teuthology.orchestra.run.smithi196.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.0.0-2160-g5a141c6290e-1bionic [3,932 B] 2020-06-04T08:22:36.843 INFO:teuthology.orchestra.run.smithi196.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.0.0-2160-g5a141c6290e-1bionic [22.4 kB] 2020-06-04T08:22:36.843 INFO:teuthology.orchestra.run.smithi196.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.0.0-2160-g5a141c6290e-1bionic [47.0 MB] 2020-06-04T08:22:39.314 INFO:teuthology.orchestra.run.smithi060.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-2160-g5a141c6290e-1bionic [19.1 kB] 2020-06-04T08:22:39.314 INFO:teuthology.orchestra.run.smithi060.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-2160-g5a141c6290e-1bionic [9,026 kB] 2020-06-04T08:22:39.834 INFO:teuthology.orchestra.run.smithi060.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-2160-g5a141c6290e-1bionic [78.9 kB] 2020-06-04T08:22:40.082 INFO:teuthology.orchestra.run.smithi196.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.0.0-2160-g5a141c6290e-1bionic [19.1 kB] 2020-06-04T08:22:40.082 INFO:teuthology.orchestra.run.smithi196.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.0.0-2160-g5a141c6290e-1bionic [9,026 kB] 2020-06-04T08:22:40.369 INFO:teuthology.orchestra.run.smithi196.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/notifications_persistent_delivery2/5a141c6290e05d95ab4ea2df3c0cb561d6b8e984/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.0.0-2160-g5a141c6290e-1bionic [78.9 kB] 2020-06-04T08:22:40.547 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 163 MB in 9s (18.7 MB/s) 2020-06-04T08:22:40.695 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-uritemplate. 2020-06-04T08:22:40.745 INFO:teuthology.orchestra.run.smithi060.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137516 files and directories currently installed.) 2020-06-04T08:22:40.748 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-06-04T08:22:40.786 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-06-04T08:22:41.066 INFO:teuthology.orchestra.run.smithi196.stdout:Fetched 163 MB in 9s (17.7 MB/s) 2020-06-04T08:22:41.188 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-06-04T08:22:41.203 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-06-04T08:22:41.215 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-uritemplate. 2020-06-04T08:22:41.262 INFO:teuthology.orchestra.run.smithi196.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137516 files and directories currently installed.) 2020-06-04T08:22:41.266 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-06-04T08:22:41.272 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-06-04T08:22:41.297 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-06-04T08:22:41.607 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../002-librbd1_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:22:41.617 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-06-04T08:22:41.632 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-06-04T08:22:41.718 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-06-04T08:22:41.736 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librbd1 (16.0.0-2160-g5a141c6290e-1bionic) over (12.2.12-0ubuntu0.18.04.5) ... 2020-06-04T08:22:42.061 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../002-librbd1_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:22:42.205 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking librbd1 (16.0.0-2160-g5a141c6290e-1bionic) over (12.2.12-0ubuntu0.18.04.5) ... 2020-06-04T08:22:42.274 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../003-librados2_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:22:42.383 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librados2 (16.0.0-2160-g5a141c6290e-1bionic) over (12.2.12-0ubuntu0.18.04.5) ... 2020-06-04T08:22:42.730 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../003-librados2_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:22:42.869 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking librados2 (16.0.0-2160-g5a141c6290e-1bionic) over (12.2.12-0ubuntu0.18.04.5) ... 2020-06-04T08:22:43.045 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libcephfs2. 2020-06-04T08:22:43.061 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:22:43.103 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libcephfs2 (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:43.406 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rados. 2020-06-04T08:22:43.419 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../005-python3-rados_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:22:43.456 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rados (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:43.488 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package libcephfs2. 2020-06-04T08:22:43.503 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../004-libcephfs2_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:22:43.546 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking libcephfs2 (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:43.784 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-06-04T08:22:43.798 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-2160-g5a141c6290e-1bionic_all.deb ... 2020-06-04T08:22:43.842 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-ceph-argparse (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:43.848 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-rados. 2020-06-04T08:22:43.863 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../005-python3-rados_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:22:43.898 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-rados (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:44.145 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cephfs. 2020-06-04T08:22:44.161 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:22:44.195 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cephfs (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:44.251 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-06-04T08:22:44.266 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../006-python3-ceph-argparse_16.0.0-2160-g5a141c6290e-1bionic_all.deb ... 2020-06-04T08:22:44.310 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-ceph-argparse (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:44.514 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-ceph-common. 2020-06-04T08:22:44.528 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-2160-g5a141c6290e-1bionic_all.deb ... 2020-06-04T08:22:44.572 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-ceph-common (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:44.612 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-cephfs. 2020-06-04T08:22:44.627 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../007-python3-cephfs_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:22:44.670 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-cephfs (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:44.884 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-prettytable. 2020-06-04T08:22:44.899 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-06-04T08:22:44.934 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-06-04T08:22:45.023 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-ceph-common. 2020-06-04T08:22:45.038 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../008-python3-ceph-common_16.0.0-2160-g5a141c6290e-1bionic_all.deb ... 2020-06-04T08:22:45.081 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-ceph-common (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:45.146 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rbd. 2020-06-04T08:22:45.161 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:22:45.186 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rbd (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:45.393 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-prettytable. 2020-06-04T08:22:45.407 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-06-04T08:22:45.460 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-06-04T08:22:45.547 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-06-04T08:22:45.563 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-06-04T08:22:45.606 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-06-04T08:22:45.712 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-rbd. 2020-06-04T08:22:45.721 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../010-python3-rbd_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:22:45.762 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-rbd (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:45.909 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-06-04T08:22:45.922 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-06-04T08:22:45.959 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-06-04T08:22:46.148 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-06-04T08:22:46.231 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-06-04T08:22:46.231 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-06-04T08:22:46.299 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package librgw2. 2020-06-04T08:22:46.311 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../013-librgw2_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:22:46.345 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librgw2 (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:46.551 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-06-04T08:22:46.565 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-06-04T08:22:46.613 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-06-04T08:22:46.881 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rgw. 2020-06-04T08:22:46.900 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:22:46.939 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rgw (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:46.979 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package librgw2. 2020-06-04T08:22:46.993 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../013-librgw2_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:22:47.037 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking librgw2 (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:47.258 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-06-04T08:22:47.272 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-06-04T08:22:47.321 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-06-04T08:22:47.581 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-rgw. 2020-06-04T08:22:47.586 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-06-04T08:22:47.595 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../014-python3-rgw_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:22:47.600 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-06-04T08:22:47.636 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-06-04T08:22:47.639 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-rgw (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:47.967 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-06-04T08:22:47.982 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-06-04T08:22:47.997 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package liboath0. 2020-06-04T08:22:48.012 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-06-04T08:22:48.025 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-06-04T08:22:48.056 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-06-04T08:22:48.303 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-06-04T08:22:48.311 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-06-04T08:22:48.353 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-06-04T08:22:48.367 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libradosstriper1. 2020-06-04T08:22:48.381 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:22:48.417 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libradosstriper1 (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:48.722 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package liboath0. 2020-06-04T08:22:48.739 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-06-04T08:22:48.781 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-06-04T08:22:48.820 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-common. 2020-06-04T08:22:48.838 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../019-ceph-common_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:22:48.867 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-common (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:49.086 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package libradosstriper1. 2020-06-04T08:22:49.097 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../018-libradosstriper1_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:22:49.134 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking libradosstriper1 (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:49.537 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package ceph-common. 2020-06-04T08:22:49.554 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../019-ceph-common_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:22:49.595 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking ceph-common (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:50.826 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-base. 2020-06-04T08:22:50.842 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../020-ceph-base_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:22:50.898 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-base (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:51.443 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package ceph-base. 2020-06-04T08:22:51.458 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../020-ceph-base_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:22:51.512 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking ceph-base (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:52.222 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-dateutil. 2020-06-04T08:22:52.235 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-06-04T08:22:52.263 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-06-04T08:22:52.563 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-dateutil. 2020-06-04T08:22:52.579 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-06-04T08:22:52.642 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-asn1crypto. 2020-06-04T08:22:52.646 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-06-04T08:22:52.656 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-06-04T08:22:52.700 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-06-04T08:22:52.939 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cffi-backend. 2020-06-04T08:22:52.952 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-06-04T08:22:52.989 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-06-04T08:22:53.033 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-asn1crypto. 2020-06-04T08:22:53.047 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-06-04T08:22:53.250 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cryptography. 2020-06-04T08:22:53.264 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.3_amd64.deb ... 2020-06-04T08:22:53.300 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-06-04T08:22:53.524 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-06-04T08:22:53.637 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-openssl. 2020-06-04T08:22:53.653 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-06-04T08:22:53.695 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-06-04T08:22:53.886 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-cffi-backend. 2020-06-04T08:22:53.903 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-06-04T08:22:53.961 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-06-04T08:22:53.989 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-06-04T08:22:54.006 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-2160-g5a141c6290e-1bionic_all.deb ... 2020-06-04T08:22:54.048 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-modules-core (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:54.226 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-cryptography. 2020-06-04T08:22:54.237 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.3_amd64.deb ... 2020-06-04T08:22:54.280 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-06-04T08:22:54.402 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-bcrypt. 2020-06-04T08:22:54.416 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-06-04T08:22:54.460 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-06-04T08:22:54.642 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-openssl. 2020-06-04T08:22:54.657 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-06-04T08:22:54.692 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-06-04T08:22:54.763 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cherrypy3. 2020-06-04T08:22:54.779 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-06-04T08:22:54.830 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-06-04T08:22:54.961 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-06-04T08:22:54.976 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_16.0.0-2160-g5a141c6290e-1bionic_all.deb ... 2020-06-04T08:22:55.020 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking ceph-mgr-modules-core (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:22:55.175 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-jwt. 2020-06-04T08:22:55.190 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-06-04T08:22:55.225 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-06-04T08:22:55.429 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-bcrypt. 2020-06-04T08:22:55.438 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-06-04T08:22:55.482 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-06-04T08:22:55.536 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-logutils. 2020-06-04T08:22:55.551 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-06-04T08:22:55.595 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-06-04T08:22:55.801 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-cherrypy3. 2020-06-04T08:22:55.815 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-06-04T08:22:55.860 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-06-04T08:22:55.881 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-markupsafe. 2020-06-04T08:22:55.897 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-06-04T08:22:55.939 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-06-04T08:22:56.179 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-jwt. 2020-06-04T08:22:56.196 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-06-04T08:22:56.238 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-06-04T08:22:56.244 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-mako. 2020-06-04T08:22:56.256 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-06-04T08:22:56.300 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-06-04T08:22:56.568 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-logutils. 2020-06-04T08:22:56.584 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-06-04T08:22:56.611 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-simplegeneric. 2020-06-04T08:22:56.624 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-06-04T08:22:56.625 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-06-04T08:22:56.661 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-06-04T08:22:56.926 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-markupsafe. 2020-06-04T08:22:56.943 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-06-04T08:22:56.972 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-singledispatch. 2020-06-04T08:22:56.985 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-06-04T08:22:56.989 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-06-04T08:22:57.031 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-06-04T08:22:57.288 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-mako. 2020-06-04T08:22:57.304 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-06-04T08:22:57.317 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-webob. 2020-06-04T08:22:57.331 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-06-04T08:22:57.346 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-06-04T08:22:57.375 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-06-04T08:22:57.676 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-simplegeneric. 2020-06-04T08:22:57.687 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-06-04T08:22:57.728 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-bs4. 2020-06-04T08:22:57.732 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-06-04T08:22:57.743 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-06-04T08:22:57.787 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-06-04T08:22:58.052 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-singledispatch. 2020-06-04T08:22:58.068 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-06-04T08:22:58.110 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-06-04T08:22:58.114 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-waitress. 2020-06-04T08:22:58.128 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-06-04T08:22:58.172 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-06-04T08:22:58.404 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-webob. 2020-06-04T08:22:58.420 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-06-04T08:22:58.462 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-06-04T08:22:58.500 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-tempita. 2020-06-04T08:22:58.517 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-06-04T08:22:58.559 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-06-04T08:22:58.807 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-bs4. 2020-06-04T08:22:58.822 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-06-04T08:22:58.857 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-06-04T08:22:58.953 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-paste. 2020-06-04T08:22:58.967 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-06-04T08:22:59.012 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-06-04T08:22:59.234 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-waitress. 2020-06-04T08:22:59.252 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-06-04T08:22:59.293 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-06-04T08:22:59.323 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-06-04T08:22:59.336 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-06-04T08:22:59.373 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-06-04T08:22:59.612 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-tempita. 2020-06-04T08:22:59.627 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-06-04T08:22:59.671 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-06-04T08:22:59.709 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pastedeploy. 2020-06-04T08:22:59.723 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-06-04T08:22:59.770 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-06-04T08:22:59.973 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-paste. 2020-06-04T08:22:59.987 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-06-04T08:23:00.032 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-06-04T08:23:00.066 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-webtest. 2020-06-04T08:23:00.076 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-06-04T08:23:00.124 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-06-04T08:23:00.351 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-06-04T08:23:00.367 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-06-04T08:23:00.409 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-06-04T08:23:00.466 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pecan. 2020-06-04T08:23:00.480 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-06-04T08:23:00.523 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-06-04T08:23:00.738 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-pastedeploy. 2020-06-04T08:23:00.754 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-06-04T08:23:00.796 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-06-04T08:23:00.861 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-werkzeug. 2020-06-04T08:23:00.878 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-06-04T08:23:00.920 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-06-04T08:23:01.099 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-webtest. 2020-06-04T08:23:01.114 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-06-04T08:23:01.160 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-06-04T08:23:01.273 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr. 2020-06-04T08:23:01.289 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:23:01.331 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:01.485 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-pecan. 2020-06-04T08:23:01.500 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-06-04T08:23:01.543 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-06-04T08:23:01.792 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mon. 2020-06-04T08:23:01.807 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:23:01.851 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mon (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:01.906 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-werkzeug. 2020-06-04T08:23:01.919 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-06-04T08:23:01.962 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-06-04T08:23:02.317 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package ceph-mgr. 2020-06-04T08:23:02.331 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../045-ceph-mgr_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:23:02.376 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking ceph-mgr (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:02.821 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package ceph-mon. 2020-06-04T08:23:02.836 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../046-ceph-mon_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:23:02.887 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking ceph-mon (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:03.505 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-osd. 2020-06-04T08:23:03.522 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:23:03.563 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-osd (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:04.532 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package ceph-osd. 2020-06-04T08:23:04.546 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../047-ceph-osd_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:23:04.590 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking ceph-osd (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:05.811 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph. 2020-06-04T08:23:05.826 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../048-ceph_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:23:05.877 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:06.196 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-fuse. 2020-06-04T08:23:06.211 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:23:06.267 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-fuse (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:06.614 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package ceph. 2020-06-04T08:23:06.626 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../048-ceph_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:23:06.677 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking ceph (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:06.695 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mds. 2020-06-04T08:23:06.710 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:23:06.753 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mds (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:06.990 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package ceph-fuse. 2020-06-04T08:23:07.005 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../049-ceph-fuse_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:23:07.048 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking ceph-fuse (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:07.295 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package cephadm. 2020-06-04T08:23:07.310 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../051-cephadm_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:23:07.369 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking cephadm (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:07.450 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package ceph-mds. 2020-06-04T08:23:07.462 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../050-ceph-mds_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:23:07.508 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking ceph-mds (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:07.689 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-06-04T08:23:07.703 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_16.0.0-2160-g5a141c6290e-1bionic_all.deb ... 2020-06-04T08:23:07.746 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-cephadm (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:08.050 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-repoze.lru. 2020-06-04T08:23:08.065 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-1_all.deb ... 2020-06-04T08:23:08.070 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package cephadm. 2020-06-04T08:23:08.085 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../051-cephadm_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:23:08.108 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-06-04T08:23:08.128 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking cephadm (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:08.402 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-routes. 2020-06-04T08:23:08.416 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../054-python3-routes_2.4.1-1_all.deb ... 2020-06-04T08:23:08.460 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-06-04T08:23:08.473 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-06-04T08:23:08.489 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_16.0.0-2160-g5a141c6290e-1bionic_all.deb ... 2020-06-04T08:23:08.531 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking ceph-mgr-cephadm (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:08.770 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-06-04T08:23:08.784 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../055-ceph-mgr-dashboard_16.0.0-2160-g5a141c6290e-1bionic_all.deb ... 2020-06-04T08:23:08.830 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-dashboard (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:08.850 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-repoze.lru. 2020-06-04T08:23:08.864 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-1_all.deb ... 2020-06-04T08:23:08.900 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-06-04T08:23:09.195 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-routes. 2020-06-04T08:23:09.211 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../054-python3-routes_2.4.1-1_all.deb ... 2020-06-04T08:23:09.255 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-06-04T08:23:09.576 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-06-04T08:23:09.591 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../055-ceph-mgr-dashboard_16.0.0-2160-g5a141c6290e-1bionic_all.deb ... 2020-06-04T08:23:09.637 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking ceph-mgr-dashboard (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:09.703 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2020-06-04T08:23:09.719 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../056-ceph-mgr-diskprediction-cloud_16.0.0-2160-g5a141c6290e-1bionic_all.deb ... 2020-06-04T08:23:09.774 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-diskprediction-cloud (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:10.139 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-numpy. 2020-06-04T08:23:10.156 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../057-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-06-04T08:23:10.197 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-06-04T08:23:10.542 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2020-06-04T08:23:10.714 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../056-ceph-mgr-diskprediction-cloud_16.0.0-2160-g5a141c6290e-1bionic_all.deb ... 2020-06-04T08:23:10.715 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking ceph-mgr-diskprediction-cloud (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:10.787 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-decorator. 2020-06-04T08:23:10.804 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../058-python3-decorator_4.1.2-1_all.deb ... 2020-06-04T08:23:10.845 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-06-04T08:23:10.895 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-numpy. 2020-06-04T08:23:10.911 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../057-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-06-04T08:23:10.953 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-06-04T08:23:11.181 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-scipy. 2020-06-04T08:23:11.196 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../059-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-06-04T08:23:11.239 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-06-04T08:23:11.567 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-decorator. 2020-06-04T08:23:11.582 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../058-python3-decorator_4.1.2-1_all.deb ... 2020-06-04T08:23:11.625 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-06-04T08:23:11.936 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-scipy. 2020-06-04T08:23:11.951 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../059-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-06-04T08:23:11.994 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-06-04T08:23:12.424 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-06-04T08:23:12.441 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-06-04T08:23:12.491 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-06-04T08:23:13.003 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-joblib. 2020-06-04T08:23:13.017 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../061-python3-joblib_0.11-1_all.deb ... 2020-06-04T08:23:13.061 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-joblib (0.11-1) ... 2020-06-04T08:23:13.136 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-06-04T08:23:13.150 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../060-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-06-04T08:23:13.194 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-06-04T08:23:13.406 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-sklearn. 2020-06-04T08:23:13.421 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../062-python3-sklearn_0.19.1-3_all.deb ... 2020-06-04T08:23:13.465 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-06-04T08:23:13.689 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-joblib. 2020-06-04T08:23:13.705 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../061-python3-joblib_0.11-1_all.deb ... 2020-06-04T08:23:13.748 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-joblib (0.11-1) ... 2020-06-04T08:23:13.996 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-06-04T08:23:14.011 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.0.0-2160-g5a141c6290e-1bionic_all.deb ... 2020-06-04T08:23:14.054 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:14.135 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-sklearn. 2020-06-04T08:23:14.148 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../062-python3-sklearn_0.19.1-3_all.deb ... 2020-06-04T08:23:14.193 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-06-04T08:23:14.750 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-06-04T08:23:14.763 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../063-ceph-mgr-diskprediction-local_16.0.0-2160-g5a141c6290e-1bionic_all.deb ... 2020-06-04T08:23:14.809 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking ceph-mgr-diskprediction-local (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:15.091 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-monotonic. 2020-06-04T08:23:15.105 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../064-python3-monotonic_1.1-2_all.deb ... 2020-06-04T08:23:15.141 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-06-04T08:23:15.435 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-fasteners. 2020-06-04T08:23:15.457 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../065-python3-fasteners_0.12.0-3_all.deb ... 2020-06-04T08:23:15.499 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-06-04T08:23:15.805 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-httplib2. 2020-06-04T08:23:15.820 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../066-python3-httplib2_0.9.2+dfsg-1ubuntu0.1_all.deb ... 2020-06-04T08:23:15.867 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2020-06-04T08:23:15.910 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-monotonic. 2020-06-04T08:23:15.927 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../064-python3-monotonic_1.1-2_all.deb ... 2020-06-04T08:23:15.969 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-06-04T08:23:16.166 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pyasn1. 2020-06-04T08:23:16.186 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3_all.deb ... 2020-06-04T08:23:16.225 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-06-04T08:23:16.329 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-fasteners. 2020-06-04T08:23:16.347 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../065-python3-fasteners_0.12.0-3_all.deb ... 2020-06-04T08:23:16.391 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-06-04T08:23:16.570 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-06-04T08:23:16.584 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-06-04T08:23:16.630 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-06-04T08:23:16.701 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-httplib2. 2020-06-04T08:23:16.714 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../066-python3-httplib2_0.9.2+dfsg-1ubuntu0.1_all.deb ... 2020-06-04T08:23:16.757 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2020-06-04T08:23:16.948 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rsa. 2020-06-04T08:23:16.965 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../069-python3-rsa_3.4.2-1_all.deb ... 2020-06-04T08:23:17.006 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-06-04T08:23:17.060 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-pyasn1. 2020-06-04T08:23:17.074 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3_all.deb ... 2020-06-04T08:23:17.118 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-06-04T08:23:17.368 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-06-04T08:23:17.384 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../070-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-06-04T08:23:17.426 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-06-04T08:23:17.446 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-06-04T08:23:17.462 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-06-04T08:23:17.505 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-06-04T08:23:17.850 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-rsa. 2020-06-04T08:23:17.864 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../069-python3-rsa_3.4.2-1_all.deb ... 2020-06-04T08:23:17.905 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-oauth2client. 2020-06-04T08:23:17.907 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-06-04T08:23:17.920 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../071-python3-oauth2client_4.1.2-3_all.deb ... 2020-06-04T08:23:17.964 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-06-04T08:23:18.252 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-06-04T08:23:18.267 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../070-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-06-04T08:23:18.300 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-setuptools. 2020-06-04T08:23:18.311 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-06-04T08:23:18.316 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../072-python3-setuptools_39.0.1-2_all.deb ... 2020-06-04T08:23:18.366 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-06-04T08:23:18.720 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-websocket. 2020-06-04T08:23:18.737 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../073-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-06-04T08:23:18.740 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-oauth2client. 2020-06-04T08:23:18.757 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../071-python3-oauth2client_4.1.2-3_all.deb ... 2020-06-04T08:23:18.778 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-06-04T08:23:18.798 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-06-04T08:23:19.098 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-kubernetes. 2020-06-04T08:23:19.113 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../074-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-06-04T08:23:19.157 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-06-04T08:23:19.178 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-setuptools. 2020-06-04T08:23:19.191 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../072-python3-setuptools_39.0.1-2_all.deb ... 2020-06-04T08:23:19.234 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-06-04T08:23:19.569 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-06-04T08:23:19.584 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.0.0-2160-g5a141c6290e-1bionic_all.deb ... 2020-06-04T08:23:19.621 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-websocket. 2020-06-04T08:23:19.627 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:19.638 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../073-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-06-04T08:23:19.679 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-06-04T08:23:19.997 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-json-pointer. 2020-06-04T08:23:20.007 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-kubernetes. 2020-06-04T08:23:20.013 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../076-python3-json-pointer_1.10-1_all.deb ... 2020-06-04T08:23:20.044 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../074-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-06-04T08:23:20.055 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-06-04T08:23:20.065 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-06-04T08:23:20.459 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-jsonpatch. 2020-06-04T08:23:20.475 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../077-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-06-04T08:23:20.487 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-06-04T08:23:20.503 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../075-ceph-mgr-k8sevents_16.0.0-2160-g5a141c6290e-1bionic_all.deb ... 2020-06-04T08:23:20.520 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-06-04T08:23:20.544 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking ceph-mgr-k8sevents (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:20.821 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-06-04T08:23:20.837 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../078-ceph-mgr-rook_16.0.0-2160-g5a141c6290e-1bionic_all.deb ... 2020-06-04T08:23:20.863 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-json-pointer. 2020-06-04T08:23:20.878 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-rook (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:20.879 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../076-python3-json-pointer_1.10-1_all.deb ... 2020-06-04T08:23:20.922 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-06-04T08:23:21.258 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libonig4:amd64. 2020-06-04T08:23:21.270 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../079-libonig4_6.7.0-1_amd64.deb ... 2020-06-04T08:23:21.275 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-jsonpatch. 2020-06-04T08:23:21.294 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../077-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-06-04T08:23:21.331 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-06-04T08:23:21.336 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-06-04T08:23:21.670 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-06-04T08:23:21.684 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../078-ceph-mgr-rook_16.0.0-2160-g5a141c6290e-1bionic_all.deb ... 2020-06-04T08:23:21.702 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libjq1:amd64. 2020-06-04T08:23:21.716 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../080-libjq1_1.5+dfsg-2_amd64.deb ... 2020-06-04T08:23:21.727 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking ceph-mgr-rook (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:21.837 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-06-04T08:23:22.080 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package libonig4:amd64. 2020-06-04T08:23:22.096 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../079-libonig4_6.7.0-1_amd64.deb ... 2020-06-04T08:23:22.109 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package jq. 2020-06-04T08:23:22.122 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../081-jq_1.5+dfsg-2_amd64.deb ... 2020-06-04T08:23:22.138 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-06-04T08:23:22.165 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-06-04T08:23:22.460 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package socat. 2020-06-04T08:23:22.491 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package libjq1:amd64. 2020-06-04T08:23:22.506 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../080-libjq1_1.5+dfsg-2_amd64.deb ... 2020-06-04T08:23:22.527 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../082-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-06-04T08:23:22.627 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-06-04T08:23:22.817 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-06-04T08:23:22.921 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package jq. 2020-06-04T08:23:22.938 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../081-jq_1.5+dfsg-2_amd64.deb ... 2020-06-04T08:23:22.988 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-06-04T08:23:23.213 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-06-04T08:23:23.228 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../083-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-06-04T08:23:23.249 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package socat. 2020-06-04T08:23:23.265 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../082-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-06-04T08:23:23.271 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-06-04T08:23:23.307 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-06-04T08:23:23.558 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package xmlstarlet. 2020-06-04T08:23:23.574 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../084-xmlstarlet_1.6.1-2_amd64.deb ... 2020-06-04T08:23:23.630 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-06-04T08:23:23.686 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-06-04T08:23:23.710 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../083-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-06-04T08:23:23.744 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-06-04T08:23:23.954 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-test. 2020-06-04T08:23:23.970 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../085-ceph-test_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:23:24.005 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package xmlstarlet. 2020-06-04T08:23:24.012 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-test (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:24.022 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../084-xmlstarlet_1.6.1-2_amd64.deb ... 2020-06-04T08:23:24.063 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-06-04T08:23:24.835 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package ceph-test. 2020-06-04T08:23:24.988 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../085-ceph-test_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:23:24.989 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking ceph-test (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:27.650 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libcephfs-dev. 2020-06-04T08:23:27.666 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../086-libcephfs-dev_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:23:27.712 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libcephfs-dev (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:28.019 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-attr. 2020-06-04T08:23:28.035 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../087-python-attr_17.4.0-2_all.deb ... 2020-06-04T08:23:28.078 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-attr (17.4.0-2) ... 2020-06-04T08:23:28.418 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-funcsigs. 2020-06-04T08:23:28.428 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../088-python-funcsigs_1.0.2-4_all.deb ... 2020-06-04T08:23:28.429 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package libcephfs-dev. 2020-06-04T08:23:28.444 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../086-libcephfs-dev_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:23:28.480 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-06-04T08:23:28.496 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking libcephfs-dev (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:28.808 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python-attr. 2020-06-04T08:23:28.828 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../087-python-attr_17.4.0-2_all.deb ... 2020-06-04T08:23:28.850 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-pluggy. 2020-06-04T08:23:28.865 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../089-python-pluggy_0.6.0-1_all.deb ... 2020-06-04T08:23:28.865 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python-attr (17.4.0-2) ... 2020-06-04T08:23:28.908 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-06-04T08:23:29.169 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python-funcsigs. 2020-06-04T08:23:29.183 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../088-python-funcsigs_1.0.2-4_all.deb ... 2020-06-04T08:23:29.211 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-py. 2020-06-04T08:23:29.226 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-06-04T08:23:29.227 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../090-python-py_1.5.2-1_all.deb ... 2020-06-04T08:23:29.269 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-py (1.5.2-1) ... 2020-06-04T08:23:29.554 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python-pluggy. 2020-06-04T08:23:29.571 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../089-python-pluggy_0.6.0-1_all.deb ... 2020-06-04T08:23:29.606 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-pytest. 2020-06-04T08:23:29.612 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-06-04T08:23:29.621 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../091-python-pytest_3.3.2-2_all.deb ... 2020-06-04T08:23:29.664 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-06-04T08:23:29.932 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python-py. 2020-06-04T08:23:29.948 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../090-python-py_1.5.2-1_all.deb ... 2020-06-04T08:23:29.998 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python-py (1.5.2-1) ... 2020-06-04T08:23:30.018 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-attr. 2020-06-04T08:23:30.034 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../092-python3-attr_17.4.0-2_all.deb ... 2020-06-04T08:23:30.076 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-06-04T08:23:30.326 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python-pytest. 2020-06-04T08:23:30.343 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../091-python-pytest_3.3.2-2_all.deb ... 2020-06-04T08:23:30.379 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cycler. 2020-06-04T08:23:30.384 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-06-04T08:23:30.397 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../093-python3-cycler_0.10.0-1_all.deb ... 2020-06-04T08:23:30.437 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-06-04T08:23:30.682 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-webencodings. 2020-06-04T08:23:30.696 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../094-python3-webencodings_0.5-2_all.deb ... 2020-06-04T08:23:30.731 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-06-04T08:23:30.779 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-attr. 2020-06-04T08:23:30.794 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../092-python3-attr_17.4.0-2_all.deb ... 2020-06-04T08:23:30.837 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-06-04T08:23:31.033 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-html5lib. 2020-06-04T08:23:31.041 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../095-python3-html5lib_0.999999999-1_all.deb ... 2020-06-04T08:23:31.085 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-06-04T08:23:31.149 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-cycler. 2020-06-04T08:23:31.163 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../093-python3-cycler_0.10.0-1_all.deb ... 2020-06-04T08:23:31.207 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-06-04T08:23:31.438 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-06-04T08:23:31.452 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../096-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-06-04T08:23:31.496 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-06-04T08:23:31.509 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-webencodings. 2020-06-04T08:23:31.525 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../094-python3-webencodings_0.5-2_all.deb ... 2020-06-04T08:23:31.568 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-06-04T08:23:31.872 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-html5lib. 2020-06-04T08:23:31.890 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../095-python3-html5lib_0.999999999-1_all.deb ... 2020-06-04T08:23:31.924 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pyparsing. 2020-06-04T08:23:31.928 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-06-04T08:23:31.940 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../097-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-06-04T08:23:31.983 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-06-04T08:23:32.315 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-06-04T08:23:32.327 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-tz. 2020-06-04T08:23:32.330 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../096-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-06-04T08:23:32.343 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../098-python3-tz_2018.3-2_all.deb ... 2020-06-04T08:23:32.382 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-06-04T08:23:32.386 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-tz (2018.3-2) ... 2020-06-04T08:23:32.638 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-matplotlib. 2020-06-04T08:23:32.655 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../099-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-06-04T08:23:32.705 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-06-04T08:23:32.835 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-pyparsing. 2020-06-04T08:23:32.850 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../097-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-06-04T08:23:32.902 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-06-04T08:23:33.229 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-tz. 2020-06-04T08:23:33.243 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../098-python3-tz_2018.3-2_all.deb ... 2020-06-04T08:23:33.290 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-tz (2018.3-2) ... 2020-06-04T08:23:33.394 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-olefile. 2020-06-04T08:23:33.409 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../100-python3-olefile_0.45.1-1_all.deb ... 2020-06-04T08:23:33.458 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-06-04T08:23:33.632 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-matplotlib. 2020-06-04T08:23:33.639 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../099-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-06-04T08:23:33.682 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-06-04T08:23:33.755 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pastescript. 2020-06-04T08:23:33.772 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../101-python3-pastescript_2.0.2-2_all.deb ... 2020-06-04T08:23:33.814 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-06-04T08:23:34.159 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pil:amd64. 2020-06-04T08:23:34.174 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../102-python3-pil_5.1.0-1ubuntu0.2_amd64.deb ... 2020-06-04T08:23:34.222 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.2) ... 2020-06-04T08:23:34.329 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-olefile. 2020-06-04T08:23:34.346 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../100-python3-olefile_0.45.1-1_all.deb ... 2020-06-04T08:23:34.388 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-06-04T08:23:34.596 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pluggy. 2020-06-04T08:23:34.611 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../103-python3-pluggy_0.6.0-1_all.deb ... 2020-06-04T08:23:34.661 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-06-04T08:23:34.730 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-pastescript. 2020-06-04T08:23:34.746 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../101-python3-pastescript_2.0.2-2_all.deb ... 2020-06-04T08:23:34.782 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-06-04T08:23:34.972 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-py. 2020-06-04T08:23:34.988 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../104-python3-py_1.5.2-1_all.deb ... 2020-06-04T08:23:35.031 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-py (1.5.2-1) ... 2020-06-04T08:23:35.118 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-pil:amd64. 2020-06-04T08:23:35.136 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../102-python3-pil_5.1.0-1ubuntu0.2_amd64.deb ... 2020-06-04T08:23:35.187 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.2) ... 2020-06-04T08:23:35.360 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pyinotify. 2020-06-04T08:23:35.374 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../105-python3-pyinotify_0.9.6-1_all.deb ... 2020-06-04T08:23:35.418 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-06-04T08:23:35.550 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-pluggy. 2020-06-04T08:23:35.564 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../103-python3-pluggy_0.6.0-1_all.deb ... 2020-06-04T08:23:35.608 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-06-04T08:23:35.737 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pytest. 2020-06-04T08:23:35.752 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../106-python3-pytest_3.3.2-2_all.deb ... 2020-06-04T08:23:35.795 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-06-04T08:23:35.895 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-py. 2020-06-04T08:23:35.912 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../104-python3-py_1.5.2-1_all.deb ... 2020-06-04T08:23:35.954 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-py (1.5.2-1) ... 2020-06-04T08:23:36.170 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-simplejson. 2020-06-04T08:23:36.180 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../107-python3-simplejson_3.13.2-1_amd64.deb ... 2020-06-04T08:23:36.224 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-06-04T08:23:36.300 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-pyinotify. 2020-06-04T08:23:36.314 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../105-python3-pyinotify_0.9.6-1_all.deb ... 2020-06-04T08:23:36.357 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-06-04T08:23:36.561 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-tk:amd64. 2020-06-04T08:23:36.580 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../108-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-06-04T08:23:36.619 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-06-04T08:23:36.684 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-pytest. 2020-06-04T08:23:36.699 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../106-python3-pytest_3.3.2-2_all.deb ... 2020-06-04T08:23:36.743 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-06-04T08:23:37.080 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-simplejson. 2020-06-04T08:23:37.094 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../107-python3-simplejson_3.13.2-1_amd64.deb ... 2020-06-04T08:23:37.115 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package radosgw. 2020-06-04T08:23:37.129 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../109-radosgw_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:23:37.138 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-06-04T08:23:37.173 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking radosgw (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:37.466 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package python3-tk:amd64. 2020-06-04T08:23:37.480 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../108-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-06-04T08:23:37.524 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-06-04T08:23:37.978 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package radosgw. 2020-06-04T08:23:37.994 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../109-radosgw_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:23:38.036 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking radosgw (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:38.152 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package rbd-fuse. 2020-06-04T08:23:38.168 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../110-rbd-fuse_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:23:38.210 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking rbd-fuse (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:38.517 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-06-04T08:23:38.853 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-06-04T08:23:38.998 INFO:teuthology.orchestra.run.smithi196.stdout:Selecting previously unselected package rbd-fuse. 2020-06-04T08:23:39.015 INFO:teuthology.orchestra.run.smithi196.stdout:Preparing to unpack .../110-rbd-fuse_16.0.0-2160-g5a141c6290e-1bionic_amd64.deb ... 2020-06-04T08:23:39.055 INFO:teuthology.orchestra.run.smithi196.stdout:Unpacking rbd-fuse (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:39.212 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-06-04T08:23:39.363 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-06-04T08:23:39.363 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-06-04T08:23:39.672 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-06-04T08:23:39.959 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.2) ... 2020-06-04T08:23:40.041 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-06-04T08:23:40.166 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-06-04T08:23:40.352 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-06-04T08:23:40.647 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-webencodings (0.5-2) ... 2020-06-04T08:23:40.769 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.2) ... 2020-06-04T08:23:40.946 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-attr (17.4.0-2) ... 2020-06-04T08:23:41.139 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-06-04T08:23:41.164 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-06-04T08:23:41.298 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-py (1.5.2-1) ... 2020-06-04T08:23:41.438 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-webencodings (0.5-2) ... 2020-06-04T08:23:41.633 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-06-04T08:23:41.723 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python-attr (17.4.0-2) ... 2020-06-04T08:23:41.899 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-06-04T08:23:42.024 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-py (1.5.2-1) ... 2020-06-04T08:23:42.032 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-06-04T08:23:42.345 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-06-04T08:23:42.378 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-06-04T08:23:42.654 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-ceph-argparse (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:42.760 INFO:teuthology.orchestra.run.smithi196.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-06-04T08:23:42.964 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-06-04T08:23:43.087 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-06-04T08:23:43.265 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-06-04T08:23:43.372 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-ceph-argparse (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:43.401 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-06-04T08:23:43.664 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-06-04T08:23:43.949 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-06-04T08:23:44.075 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-06-04T08:23:44.346 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-06-04T08:23:44.681 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-06-04T08:23:44.982 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-py (1.5.2-1) ... 2020-06-04T08:23:45.021 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-06-04T08:23:45.242 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-06-04T08:23:45.314 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-06-04T08:23:45.573 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python-py (1.5.2-1) ... 2020-06-04T08:23:45.585 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-06-04T08:23:45.674 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-06-04T08:23:45.774 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-06-04T08:23:45.937 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-06-04T08:23:46.063 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-06-04T08:23:46.100 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-06-04T08:23:46.198 INFO:teuthology.orchestra.run.smithi196.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-06-04T08:23:46.266 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-06-04T08:23:46.427 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-06-04T08:23:46.566 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-06-04T08:23:46.578 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-06-04T08:23:46.779 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-06-04T08:23:46.882 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-06-04T08:23:47.074 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-06-04T08:23:47.184 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-06-04T08:23:47.380 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-06-04T08:23:47.569 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-06-04T08:23:47.648 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-06-04T08:23:47.912 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-06-04T08:23:48.008 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-06-04T08:23:48.213 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up cephadm (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:48.346 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-06-04T08:23:48.644 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up cephadm (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:48.857 INFO:teuthology.orchestra.run.smithi060.stdout:Adding system user cephadm....done 2020-06-04T08:23:49.028 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-06-04T08:23:49.270 INFO:teuthology.orchestra.run.smithi196.stdout:Adding system user cephadm....done 2020-06-04T08:23:49.347 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-06-04T08:23:49.425 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-06-04T08:23:49.498 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-06-04T08:23:49.726 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-06-04T08:23:49.853 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-06-04T08:23:49.857 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-06-04T08:23:50.158 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-06-04T08:23:50.197 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-06-04T08:23:50.284 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-06-04T08:23:50.497 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-06-04T08:23:50.613 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-06-04T08:23:50.628 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-ceph-common (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:50.924 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-ceph-common (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:23:50.963 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-monotonic (1.1-2) ... 2020-06-04T08:23:51.199 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-monotonic (1.1-2) ... 2020-06-04T08:23:51.258 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-06-04T08:23:51.406 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-06-04T08:23:51.475 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-06-04T08:23:51.601 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-06-04T08:23:51.758 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-06-04T08:23:51.848 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-06-04T08:23:51.888 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-06-04T08:23:51.970 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-06-04T08:23:52.059 INFO:teuthology.orchestra.run.smithi196.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-06-04T08:23:52.100 INFO:teuthology.orchestra.run.smithi196.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-06-04T08:23:52.135 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-06-04T08:23:52.346 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-06-04T08:23:52.445 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2020-06-04T08:23:52.639 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.1) ... 2020-06-04T08:23:52.750 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-06-04T08:23:52.923 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-06-04T08:23:53.089 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-attr (17.4.0-2) ... 2020-06-04T08:23:53.241 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-attr (17.4.0-2) ... 2020-06-04T08:23:53.542 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-06-04T08:23:53.735 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-06-04T08:23:53.748 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-06-04T08:23:53.941 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-06-04T08:23:54.145 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-06-04T08:23:54.235 INFO:teuthology.orchestra.run.smithi196.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-06-04T08:23:54.359 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-06-04T08:23:54.449 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-06-04T08:23:54.488 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up liboath0 (2.6.1-1) ... 2020-06-04T08:23:54.639 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-06-04T08:23:54.720 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up liboath0 (2.6.1-1) ... 2020-06-04T08:23:54.740 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-06-04T08:23:54.913 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-06-04T08:23:54.975 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-06-04T08:23:55.047 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-06-04T08:23:55.340 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-06-04T08:23:55.418 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-06-04T08:23:55.466 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-06-04T08:23:55.819 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-06-04T08:23:55.859 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-06-04T08:23:55.945 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-06-04T08:23:55.986 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-06-04T08:23:56.254 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-tz (2018.3-2) ... 2020-06-04T08:23:56.296 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-tz (2018.3-2) ... 2020-06-04T08:23:56.547 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-06-04T08:23:56.597 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-06-04T08:23:56.823 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-06-04T08:23:56.906 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-06-04T08:23:57.116 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-joblib (0.11-1) ... 2020-06-04T08:23:57.232 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-joblib (0.11-1) ... 2020-06-04T08:23:57.476 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-routes (2.4.1-1) ... 2020-06-04T08:23:57.609 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-routes (2.4.1-1) ... 2020-06-04T08:23:57.777 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-06-04T08:23:57.910 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-06-04T08:23:58.086 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-06-04T08:23:58.211 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-06-04T08:23:58.412 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-06-04T08:23:58.554 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-06-04T08:23:58.789 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-06-04T08:23:58.923 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-06-04T08:23:59.065 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up jq (1.5+dfsg-2) ... 2020-06-04T08:23:59.174 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-06-04T08:23:59.232 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up jq (1.5+dfsg-2) ... 2020-06-04T08:23:59.358 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-06-04T08:24:00.363 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-06-04T08:24:00.564 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-06-04T08:24:00.656 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-06-04T08:24:00.916 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-06-04T08:24:00.957 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-06-04T08:24:01.192 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-06-04T08:24:01.242 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-06-04T08:24:01.368 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-06-04T08:24:01.577 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python-pytest (3.3.2-2) ... 2020-06-04T08:24:01.754 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-pytest (3.3.2-2) ... 2020-06-04T08:24:01.853 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-06-04T08:24:01.979 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-06-04T08:24:02.014 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-06-04T08:24:02.098 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up librados2 (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:02.139 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-06-04T08:24:02.197 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-06-04T08:24:02.281 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librados2 (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:02.424 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-06-04T08:24:02.507 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up libcephfs2 (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:02.608 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up libcephfs-dev (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:02.734 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libcephfs2 (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:02.742 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-rados (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:02.860 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libcephfs-dev (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:02.868 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-06-04T08:24:03.003 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rados (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:03.138 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-06-04T08:24:03.186 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-06-04T08:24:03.456 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-06-04T08:24:03.487 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-06-04T08:24:03.774 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-06-04T08:24:03.847 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-06-04T08:24:04.084 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-06-04T08:24:04.675 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-06-04T08:24:04.937 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-06-04T08:24:05.504 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-06-04T08:24:05.792 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-06-04T08:24:05.849 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up libradosstriper1 (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:05.975 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-cephfs (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:06.170 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libradosstriper1 (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:06.276 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up librgw2 (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:06.296 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cephfs (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:06.402 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up ceph-fuse (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:06.589 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librgw2 (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:06.625 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-06-04T08:24:06.625 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-06-04T08:24:06.749 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-fuse (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:06.920 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up librbd1 (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:06.984 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-06-04T08:24:06.984 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-06-04T08:24:07.038 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-06-04T08:24:07.111 INFO:teuthology.orchestra.run.smithi196.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-06-04T08:24:07.141 INFO:teuthology.orchestra.run.smithi196.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-06-04T08:24:07.266 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librbd1 (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:07.393 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-06-04T08:24:07.441 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-06-04T08:24:07.482 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-06-04T08:24:07.515 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-06-04T08:24:07.821 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-06-04T08:24:08.178 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up ceph-mgr-modules-core (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:08.294 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-06-04T08:24:08.499 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-modules-core (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:08.621 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-rgw (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:08.625 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-06-04T08:24:08.730 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up rbd-fuse (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:08.856 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up python3-rbd (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:08.934 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rgw (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:08.965 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up ceph-common (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:09.061 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up rbd-fuse (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:09.187 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rbd (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:09.314 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-common (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:09.460 INFO:teuthology.orchestra.run.smithi196.stdout:Adding group ceph....done 2020-06-04T08:24:09.819 INFO:teuthology.orchestra.run.smithi196.stdout:Adding system user ceph....done 2020-06-04T08:24:09.856 INFO:teuthology.orchestra.run.smithi060.stdout:Adding group ceph....done 2020-06-04T08:24:09.995 INFO:teuthology.orchestra.run.smithi196.stdout:Setting system user ceph properties....done 2020-06-04T08:24:10.145 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-06-04T08:24:10.277 INFO:teuthology.orchestra.run.smithi060.stdout:Adding system user ceph....done 2020-06-04T08:24:10.372 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-06-04T08:24:10.470 INFO:teuthology.orchestra.run.smithi060.stdout:Setting system user ceph properties....done 2020-06-04T08:24:10.636 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-06-04T08:24:10.668 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up radosgw (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:10.866 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-06-04T08:24:10.933 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-06-04T08:24:10.933 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-06-04T08:24:11.142 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up radosgw (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:11.353 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up ceph-test (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:11.413 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-06-04T08:24:11.413 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-06-04T08:24:11.496 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up ceph-base (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:11.827 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-test (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:11.835 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-06-04T08:24:11.953 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-base (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:12.131 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up ceph-mgr (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:12.296 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-06-04T08:24:12.328 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-06-04T08:24:12.328 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-06-04T08:24:12.572 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:12.599 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:12.759 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up ceph-osd (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:12.782 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-06-04T08:24:12.783 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-06-04T08:24:13.032 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-diskprediction-local (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:13.150 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-osd (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:13.212 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-06-04T08:24:13.212 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-06-04T08:24:13.469 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up ceph-mgr-cephadm (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:13.583 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-06-04T08:24:13.584 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-06-04T08:24:13.611 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up ceph-mgr-diskprediction-cloud (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:13.757 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up ceph-mgr-rook (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:13.854 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-cephadm (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:13.897 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up ceph-mds (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:13.997 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-diskprediction-cloud (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:14.090 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-06-04T08:24:14.091 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-06-04T08:24:14.139 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-rook (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:14.257 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mds (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:14.357 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up ceph-mon (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:14.466 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-06-04T08:24:14.467 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-06-04T08:24:14.553 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-06-04T08:24:14.553 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-06-04T08:24:14.742 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mon (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:14.833 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up ceph-mgr-k8sevents (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:14.954 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-06-04T08:24:14.955 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-06-04T08:24:14.959 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up ceph-mgr-dashboard (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:15.102 INFO:teuthology.orchestra.run.smithi196.stdout:Setting up ceph (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:15.219 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-k8sevents (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:15.228 INFO:teuthology.orchestra.run.smithi196.stdout:Processing triggers for systemd (237-3ubuntu10.39) ... 2020-06-04T08:24:15.354 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-dashboard (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:15.430 INFO:teuthology.orchestra.run.smithi196.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-06-04T08:24:15.496 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph (16.0.0-2160-g5a141c6290e-1bionic) ... 2020-06-04T08:24:15.623 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for systemd (237-3ubuntu10.39) ... 2020-06-04T08:24:15.832 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-06-04T08:24:16.759 INFO:teuthology.orchestra.run.smithi196.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-06-04T08:24:16.802 INFO:teuthology.orchestra.run.smithi196.stdout:ureadahead will be reprofiled on next reboot 2020-06-04T08:24:16.851 INFO:teuthology.orchestra.run.smithi196.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2020-06-04T08:24:17.269 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-06-04T08:24:17.312 INFO:teuthology.orchestra.run.smithi060.stdout:ureadahead will be reprofiled on next reboot 2020-06-04T08:24:17.323 INFO:teuthology.orchestra.run.smithi196.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-06-04T08:24:17.325 DEBUG:teuthology.parallel:result is None 2020-06-04T08:24:17.370 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for libc-bin (2.27-3ubuntu1) ... 2020-06-04T08:24:17.818 INFO:teuthology.orchestra.run.smithi060.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-06-04T08:24:17.821 DEBUG:teuthology.parallel:result is None 2020-06-04T08:24:17.822 INFO:teuthology.orchestra.run.smithi060:> sudo lsb_release -is 2020-06-04T08:24:17.873 INFO:teuthology.orchestra.run.smithi060.stdout:Ubuntu 2020-06-04T08:24:17.878 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-06-04T08:24:17.879 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=5a141c6290e05d95ab4ea2df3c0cb561d6b8e984 2020-06-04T08:24:18.052 INFO:teuthology.orchestra.run.smithi060:> dpkg-query -W -f '${Version}' ceph 2020-06-04T08:24:18.076 INFO:teuthology.orchestra.run.smithi060.stdout:16.0.0-2160-g5a141c6290e-1bionic 2020-06-04T08:24:18.077 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-2160-g5a141c6290e-1bionic 2020-06-04T08:24:18.078 INFO:teuthology.task.install:The correct ceph version 16.0.0-2160-g5a141c6290e-1bionic is installed. 2020-06-04T08:24:18.078 INFO:teuthology.orchestra.run.smithi196:> sudo lsb_release -is 2020-06-04T08:24:18.133 INFO:teuthology.orchestra.run.smithi196.stdout:Ubuntu 2020-06-04T08:24:18.139 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-06-04T08:24:18.139 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=5a141c6290e05d95ab4ea2df3c0cb561d6b8e984 2020-06-04T08:24:18.317 INFO:teuthology.orchestra.run.smithi196:> dpkg-query -W -f '${Version}' ceph 2020-06-04T08:24:18.341 INFO:teuthology.orchestra.run.smithi196.stdout:16.0.0-2160-g5a141c6290e-1bionic 2020-06-04T08:24:18.342 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-2160-g5a141c6290e-1bionic 2020-06-04T08:24:18.343 INFO:teuthology.task.install:The correct ceph version 16.0.0-2160-g5a141c6290e-1bionic is installed. 2020-06-04T08:24:18.343 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-06-04T08:24:18.344 INFO:teuthology.orchestra.run.smithi060:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-06-04T08:24:18.360 INFO:teuthology.orchestra.run.smithi196:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-06-04T08:24:18.403 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-06-04T08:24:18.404 INFO:teuthology.orchestra.run.smithi060:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-06-04T08:24:18.420 INFO:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-06-04T08:24:18.475 INFO:teuthology.orchestra.run.smithi196:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-06-04T08:24:18.492 INFO:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-06-04T08:24:18.549 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-06-04T08:24:18.550 INFO:teuthology.orchestra.run.smithi060:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-06-04T08:24:18.607 INFO:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-06-04T08:24:18.670 INFO:teuthology.orchestra.run.smithi196:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-06-04T08:24:18.717 INFO:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-06-04T08:24:18.779 INFO:teuthology.run_tasks:Running task ceph... 2020-06-04T08:24:18.858 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-06-04T08:24:18.859 INFO:teuthology.orchestra.run.smithi060:> sudo chmod 777 /var/log/ceph 2020-06-04T08:24:18.863 INFO:teuthology.orchestra.run.smithi196:> sudo chmod 777 /var/log/ceph 2020-06-04T08:24:18.878 INFO:tasks.ceph:Disabling ceph logrotate... 2020-06-04T08:24:18.878 INFO:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/logrotate.d/ceph 2020-06-04T08:24:18.919 INFO:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/logrotate.d/ceph 2020-06-04T08:24:18.934 INFO:tasks.ceph:Creating extra log directories... 2020-06-04T08:24:18.935 INFO:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-06-04T08:24:18.975 INFO:teuthology.orchestra.run.smithi196:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-06-04T08:24:18.990 INFO:tasks.ceph:Creating ceph cluster c1... 2020-06-04T08:24:18.991 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-06-04T08:24:18.991 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuvalif-2020-06-04_08:07:49-rgw:multisite-notifications_persistent_delivery2-distro-basic-smithi/5116651', 'branch': 'notifications_persistent_delivery2', 'description': 'rgw:multisite/{clusters.yaml frontend/beast.yaml omap_limits.yaml overrides.yaml realms/three-zone-plus-pubsub.yaml tasks/test_multi.yaml valgrind.yaml}', 'email': None, 'first_in_suite': False, 'job_id': '5116651', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuvalif-2020-06-04_08:07:49-rgw:multisite-notifications_persistent_delivery2-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'notifications_persistent_delivery2'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': '5a141c6290e05d95ab4ea2df3c0cb561d6b8e984', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '5a141c6290e05d95ab4ea2df3c0cb561d6b8e984'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'notifications_persistent_delivery2', 'sha1': '5a141c6290e05d95ab4ea2df3c0cb561d6b8e984'}}, 'owner': 'scheduled_yuvalif@teuthology', 'priority': 10, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '5a141c6290e05d95ab4ea2df3c0cb561d6b8e984', 'sleep_before_teardown': 0, 'suite': 'rgw:multisite', 'suite_branch': 'notifications_persistent_delivery2', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_notifications_persistent_delivery2/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '5a141c6290e05d95ab4ea2df3c0cb561d6b8e984', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': '5a141c6290e05d95ab4ea2df3c0cb561d6b8e984', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'tube': 'smithi', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.27832', 'targets': {'smithi060.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2sBIojSjMzey9aOmMk+qXyf+3XXUKYC8z2s2P34SimG2IxQrlBKbYii8YwiPqmDxwHXFwGe6emOAHi2Qbt5swScPnApnwLaqCisyP+CRaR48Oe1PgL/nV054P9J1dYeMbKnVdGwSABk6R3cM6e5Na3et3aDMH3L/ftTdxo4wToKWEbQCCqpgK4s2TEbKtGQ6CBuNtHtqBoV65ni853lz6jOLNfIc1RDcaNCeMeJsUrtggxjrQ8j/wxjjHqzw4PbszWwbv8sF6WBsFKQFcoA4CYliBVWzL5/BDK+FTMk0kAG4YX3FV4YYH+yPBFuAxLBqdkx5nfegBuP4xHSHpJ31b', 'smithi196.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDejCVS35mBeiMVuY30POoD2qFLttOeNDZSkW0Byb9L8vUr1j0Thf72EQ2VqZ0FKgkgkADDw2bY7nNUZYxRfjTGLJNwr66E8oNgT82C44Muwli1+YU7JIkaFAVNPxrE28N1dOjdApLLT9i0W8pQOifHPKlvMWoAPMm6V8U5RDbTWdQ6gV0oq1SPlO+PNgCmV8UG7YQ02w/YXJcRgS0mNGRGqSJi1UMaVwQsZvNv4+OI6hOzjP5kxmpt7k2ThLv4CmmlzHfE6Pzb/Tv7rw9R4Coq/EFsTuzSbt2NShPUbgNX66BDX7kNZmYt8p372zyYfQZGUjDzyXXDC1eSuNiBJVhn'}} 2020-06-04T08:24:18.992 INFO:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-06-04T08:24:19.031 INFO:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2020-06-04T08:24:19.038 INFO:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/run/ceph 2020-06-04T08:24:19.079 INFO:teuthology.orchestra.run.smithi196:> sudo install -d -m0777 -- /var/run/ceph 2020-06-04T08:24:19.141 DEBUG:teuthology.orchestra.remote:smithi060:/scratch_devs is 72B 2020-06-04T08:24:19.153 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-06-04T08:24:19.153 INFO:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-06-04T08:24:19.169 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-06-04T08:24:19.170 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-04T08:24:19.170 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 341 Links: 1 2020-06-04T08:24:19.170 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-04T08:24:19.170 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2020-06-04 08:21:56.698153016 +0000 2020-06-04T08:24:19.170 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2020-06-04 08:21:56.602201018 +0000 2020-06-04T08:24:19.171 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2020-06-04 08:21:56.602201018 +0000 2020-06-04T08:24:19.171 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2020-06-04T08:24:19.179 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2020-06-04T08:24:19.179 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2020-06-04T08:24:19.179 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000251133 s, 2.0 MB/s 2020-06-04T08:24:19.184 INFO:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-06-04T08:24:19.236 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-06-04T08:24:19.237 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-04T08:24:19.237 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 348 Links: 1 2020-06-04T08:24:19.237 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-04T08:24:19.238 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2020-06-04 08:21:56.910047010 +0000 2020-06-04T08:24:19.238 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2020-06-04 08:21:56.802101014 +0000 2020-06-04T08:24:19.238 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2020-06-04 08:21:56.802101014 +0000 2020-06-04T08:24:19.238 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2020-06-04T08:24:19.245 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2020-06-04T08:24:19.246 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2020-06-04T08:24:19.246 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000286073 s, 1.8 MB/s 2020-06-04T08:24:19.249 INFO:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-06-04T08:24:19.300 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-06-04T08:24:19.300 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-04T08:24:19.301 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 354 Links: 1 2020-06-04T08:24:19.301 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-04T08:24:19.301 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2020-06-04 08:21:57.137933006 +0000 2020-06-04T08:24:19.301 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2020-06-04 08:21:57.025989009 +0000 2020-06-04T08:24:19.301 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2020-06-04 08:21:57.025989009 +0000 2020-06-04T08:24:19.302 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2020-06-04T08:24:19.309 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2020-06-04T08:24:19.309 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2020-06-04T08:24:19.309 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.00026945 s, 1.9 MB/s 2020-06-04T08:24:19.312 INFO:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-06-04T08:24:19.360 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-06-04T08:24:19.361 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-04T08:24:19.361 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 6h/6d Inode: 360 Links: 1 2020-06-04T08:24:19.361 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-04T08:24:19.362 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2020-06-04 08:21:57.373815001 +0000 2020-06-04T08:24:19.362 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2020-06-04 08:21:57.253875003 +0000 2020-06-04T08:24:19.362 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2020-06-04 08:21:57.253875003 +0000 2020-06-04T08:24:19.362 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: - 2020-06-04T08:24:19.374 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2020-06-04T08:24:19.374 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2020-06-04T08:24:19.375 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000286944 s, 1.8 MB/s 2020-06-04T08:24:19.375 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-06-04T08:24:19.375 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-06-04T08:24:19.376 INFO:tasks.ceph:Generating config... 2020-06-04T08:24:19.377 INFO:tasks.ceph:[client] debug rgw = 20 2020-06-04T08:24:19.377 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-06-04T08:24:19.378 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-06-04T08:24:19.378 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-06-04T08:24:19.379 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-06-04T08:24:19.379 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-06-04T08:24:19.379 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-06-04T08:24:19.379 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-06-04T08:24:19.380 INFO:tasks.ceph:[client] setgroup = ceph 2020-06-04T08:24:19.380 INFO:tasks.ceph:[client] setuser = ceph 2020-06-04T08:24:19.380 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-06-04T08:24:19.381 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-06-04T08:24:19.381 INFO:tasks.ceph:[mgr] debug ms = 1 2020-06-04T08:24:19.381 INFO:tasks.ceph:[mon] debug mon = 20 2020-06-04T08:24:19.381 INFO:tasks.ceph:[mon] debug ms = 1 2020-06-04T08:24:19.382 INFO:tasks.ceph:[mon] debug paxos = 20 2020-06-04T08:24:19.382 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-06-04T08:24:19.382 INFO:tasks.ceph:[osd] debug filestore = 20 2020-06-04T08:24:19.382 INFO:tasks.ceph:[osd] debug journal = 20 2020-06-04T08:24:19.383 INFO:tasks.ceph:[osd] debug ms = 20 2020-06-04T08:24:19.383 INFO:tasks.ceph:[osd] debug osd = 25 2020-06-04T08:24:19.385 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-06-04T08:24:19.386 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-06-04T08:24:19.386 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-06-04T08:24:19.386 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-06-04T08:24:19.387 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-06-04T08:24:19.387 INFO:tasks.ceph:Setting up c1.mon.a... 2020-06-04T08:24:19.388 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2020-06-04T08:24:19.450 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/c1.keyring 2020-06-04T08:24:19.453 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2020-06-04T08:24:19.534 INFO:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/c1.keyring 2020-06-04T08:24:19.590 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.60')] 2020-06-04T08:24:19.591 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.60 --print /home/ubuntu/cephtest/c1.monmap 2020-06-04T08:24:19.664 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2020-06-04T08:24:19.664 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: generated fsid 959b9e79-8d6d-4cc9-b316-d7687f41bb88 2020-06-04T08:24:19.665 INFO:teuthology.orchestra.run.smithi060.stdout:epoch 0 2020-06-04T08:24:19.665 INFO:teuthology.orchestra.run.smithi060.stdout:fsid 959b9e79-8d6d-4cc9-b316-d7687f41bb88 2020-06-04T08:24:19.665 INFO:teuthology.orchestra.run.smithi060.stdout:last_changed 2020-06-04T08:24:19.663322+0000 2020-06-04T08:24:19.666 INFO:teuthology.orchestra.run.smithi060.stdout:created 2020-06-04T08:24:19.663322+0000 2020-06-04T08:24:19.666 INFO:teuthology.orchestra.run.smithi060.stdout:min_mon_release 0 (unknown) 2020-06-04T08:24:19.666 INFO:teuthology.orchestra.run.smithi060.stdout:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2020-06-04T08:24:19.666 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2020-06-04T08:24:19.668 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 959b9e79-8d6d-4cc9-b316-d7687f41bb88... 2020-06-04T08:24:19.669 INFO:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-06-04T08:24:19.712 INFO:teuthology.orchestra.run.smithi196:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2020-06-04T08:24:19.743 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2020-06-04T08:24:19.743 INFO:teuthology.orchestra.run.smithi060.stdout: chdir = "" 2020-06-04T08:24:19.743 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-06-04T08:24:19.744 INFO:teuthology.orchestra.run.smithi060.stdout: auth supported = cephx 2020-06-04T08:24:19.744 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.744 INFO:teuthology.orchestra.run.smithi060.stdout: filestore xattr use omap = true 2020-06-04T08:24:19.744 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.744 INFO:teuthology.orchestra.run.smithi060.stdout: mon clock drift allowed = 1.000 2020-06-04T08:24:19.745 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.745 INFO:teuthology.orchestra.run.smithi060.stdout: osd crush chooseleaf type = 0 2020-06-04T08:24:19.745 INFO:teuthology.orchestra.run.smithi060.stdout: auth debug = true 2020-06-04T08:24:19.745 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.747 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on old message = true 2020-06-04T08:24:19.748 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on bug = true 2020-06-04T08:24:19.748 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.748 INFO:teuthology.orchestra.run.smithi060.stdout: mon max pg per osd = 10000 # >= luminous 2020-06-04T08:24:19.749 INFO:teuthology.orchestra.run.smithi060.stdout: mon pg warn max object skew = 0 2020-06-04T08:24:19.749 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.749 INFO:teuthology.orchestra.run.smithi060.stdout: # disable pg_autoscaler by default for new pools 2020-06-04T08:24:19.749 INFO:teuthology.orchestra.run.smithi060.stdout: osd_pool_default_pg_autoscale_mode = off 2020-06-04T08:24:19.750 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.750 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default size = 2 2020-06-04T08:24:19.750 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.750 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow primary affinity = true 2020-06-04T08:24:19.750 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow pg remap = true 2020-06-04T08:24:19.751 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on legacy crush tunables = false 2020-06-04T08:24:19.751 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on crush straw calc version zero = false 2020-06-04T08:24:19.751 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on no sortbitwise = false 2020-06-04T08:24:19.751 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on osd down out interval zero = false 2020-06-04T08:24:19.752 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on too few osds = false 2020-06-04T08:24:19.752 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-06-04T08:24:19.752 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_no_redundancy = false 2020-06-04T08:24:19.752 INFO:teuthology.orchestra.run.smithi060.stdout: mon_allow_pool_size_one = true 2020-06-04T08:24:19.752 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.753 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-06-04T08:24:19.753 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.753 INFO:teuthology.orchestra.run.smithi060.stdout: osd default data pool replay window = 5 2020-06-04T08:24:19.753 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.753 INFO:teuthology.orchestra.run.smithi060.stdout: mon allow pool delete = true 2020-06-04T08:24:19.754 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.754 INFO:teuthology.orchestra.run.smithi060.stdout: mon cluster log file level = debug 2020-06-04T08:24:19.754 INFO:teuthology.orchestra.run.smithi060.stdout: debug asserts on shutdown = true 2020-06-04T08:24:19.754 INFO:teuthology.orchestra.run.smithi060.stdout: mon host = 172.21.15.60 2020-06-04T08:24:19.754 INFO:teuthology.orchestra.run.smithi060.stdout: osd heartbeat grace = 40 2020-06-04T08:24:19.755 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = 959b9e79-8d6d-4cc9-b316-d7687f41bb88 2020-06-04T08:24:19.760 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.760 INFO:teuthology.orchestra.run.smithi060.stdout:[osd] 2020-06-04T08:24:19.760 INFO:teuthology.orchestra.run.smithi060.stdout: osd journal size = 100 2020-06-04T08:24:19.761 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.761 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub load threshold = 5.0 2020-06-04T08:24:19.761 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub max interval = 600 2020-06-04T08:24:19.761 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.761 INFO:teuthology.orchestra.run.smithi060.stdout: osd recover clone overlap = true 2020-06-04T08:24:19.762 INFO:teuthology.orchestra.run.smithi060.stdout: osd recovery max chunk = 1048576 2020-06-04T08:24:19.762 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.762 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug shutdown = true 2020-06-04T08:24:19.763 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug op order = true 2020-06-04T08:24:19.763 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify stray on activate = true 2020-06-04T08:24:19.763 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.764 INFO:teuthology.orchestra.run.smithi060.stdout: osd open classes on start = true 2020-06-04T08:24:19.764 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug pg log writeout = true 2020-06-04T08:24:19.764 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.764 INFO:teuthology.orchestra.run.smithi060.stdout: osd deep scrub update digest min age = 30 2020-06-04T08:24:19.765 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.765 INFO:teuthology.orchestra.run.smithi060.stdout: osd map max advance = 10 2020-06-04T08:24:19.765 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.765 INFO:teuthology.orchestra.run.smithi060.stdout: journal zero on create = true 2020-06-04T08:24:19.765 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.766 INFO:teuthology.orchestra.run.smithi060.stdout: filestore ondisk finisher threads = 3 2020-06-04T08:24:19.766 INFO:teuthology.orchestra.run.smithi060.stdout: filestore apply finisher threads = 3 2020-06-04T08:24:19.766 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.766 INFO:teuthology.orchestra.run.smithi060.stdout: bdev debug aio = true 2020-06-04T08:24:19.766 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug misdirected ops = true 2020-06-04T08:24:19.767 INFO:teuthology.orchestra.run.smithi060.stdout: debug filestore = 20 2020-06-04T08:24:19.767 INFO:teuthology.orchestra.run.smithi060.stdout: debug journal = 20 2020-06-04T08:24:19.767 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 20 2020-06-04T08:24:19.767 INFO:teuthology.orchestra.run.smithi060.stdout: debug osd = 25 2020-06-04T08:24:19.768 INFO:teuthology.orchestra.run.smithi060.stdout: osd fast shutdown = False 2020-06-04T08:24:19.768 INFO:teuthology.orchestra.run.smithi060.stdout: rocksdb delete range threshold = 0 2020-06-04T08:24:19.768 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.768 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr] 2020-06-04T08:24:19.768 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2020-06-04T08:24:19.769 INFO:teuthology.orchestra.run.smithi060.stdout: debug mgr = 20 2020-06-04T08:24:19.769 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2020-06-04T08:24:19.769 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2020-06-04T08:24:19.786 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2020-06-04T08:24:19.787 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2020-06-04T08:24:19.793 INFO:teuthology.orchestra.run.smithi060.stdout: mgr/telemetry/nag = false 2020-06-04T08:24:19.794 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.796 INFO:teuthology.orchestra.run.smithi060.stdout:[mon] 2020-06-04T08:24:19.796 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2020-06-04T08:24:19.797 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2020-06-04T08:24:19.797 INFO:teuthology.orchestra.run.smithi060.stdout: debug paxos = 20 2020-06-04T08:24:19.797 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2020-06-04T08:24:19.798 INFO:teuthology.orchestra.run.smithi060.stdout: mon data avail warn = 5 2020-06-04T08:24:19.798 INFO:teuthology.orchestra.run.smithi060.stdout: mon mgr mkfs grace = 240 2020-06-04T08:24:19.798 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2020-06-04T08:24:19.801 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd reporter subtree level = osd 2020-06-04T08:24:19.802 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd prime pg temp = true 2020-06-04T08:24:19.802 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2020-06-04T08:24:19.802 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd crush smoke test = False 2020-06-04T08:24:19.802 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:19.803 INFO:teuthology.orchestra.run.smithi060.stdout:[client] 2020-06-04T08:24:19.803 INFO:teuthology.orchestra.run.smithi060.stdout: rgw cache enabled = true 2020-06-04T08:24:19.803 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable ops log = true 2020-06-04T08:24:19.803 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable usage log = true 2020-06-04T08:24:19.803 INFO:teuthology.orchestra.run.smithi060.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-06-04T08:24:19.804 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-06-04T08:24:19.804 INFO:teuthology.orchestra.run.smithi060.stdout: debug rgw = 20 2020-06-04T08:24:19.804 INFO:teuthology.orchestra.run.smithi060.stdout: rgw crypt require ssl = False 2020-06-04T08:24:19.804 INFO:teuthology.orchestra.run.smithi060.stdout: rgw crypt s3 kms backend = testing 2020-06-04T08:24:19.804 INFO:teuthology.orchestra.run.smithi060.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-06-04T08:24:19.805 INFO:teuthology.orchestra.run.smithi060.stdout: rgw curl low speed time = 300 2020-06-04T08:24:19.805 INFO:teuthology.orchestra.run.smithi060.stdout: rgw data log num shards = 4 2020-06-04T08:24:19.805 INFO:teuthology.orchestra.run.smithi060.stdout: rgw md log max shards = 4 2020-06-04T08:24:19.807 INFO:teuthology.orchestra.run.smithi060.stdout: rgw sync log trim interval = 0 2020-06-04T08:24:19.807 INFO:teuthology.orchestra.run.smithi060.stdout: setgroup = ceph 2020-06-04T08:24:19.807 INFO:teuthology.orchestra.run.smithi060.stdout: setuser = ceph 2020-06-04T08:24:19.807 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.a] 2020-06-04T08:24:19.807 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.0] 2020-06-04T08:24:19.808 INFO:teuthology.orchestra.run.smithi060.stdout: osd_max_omap_entries_per_request = 10 2020-06-04T08:24:19.808 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.1] 2020-06-04T08:24:19.808 INFO:teuthology.orchestra.run.smithi060.stdout: osd_max_omap_entries_per_request = 1000 2020-06-04T08:24:19.808 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.2] 2020-06-04T08:24:19.808 INFO:teuthology.orchestra.run.smithi060.stdout: osd_max_omap_entries_per_request = 10000 2020-06-04T08:24:19.810 INFO:teuthology.orchestra.run.smithi196.stdout:[global] 2020-06-04T08:24:19.810 INFO:teuthology.orchestra.run.smithi196.stdout: chdir = "" 2020-06-04T08:24:19.810 INFO:teuthology.orchestra.run.smithi196.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-06-04T08:24:19.810 INFO:teuthology.orchestra.run.smithi196.stdout: auth supported = cephx 2020-06-04T08:24:19.811 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.811 INFO:teuthology.orchestra.run.smithi196.stdout: filestore xattr use omap = true 2020-06-04T08:24:19.811 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.811 INFO:teuthology.orchestra.run.smithi196.stdout: mon clock drift allowed = 1.000 2020-06-04T08:24:19.811 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.811 INFO:teuthology.orchestra.run.smithi196.stdout: osd crush chooseleaf type = 0 2020-06-04T08:24:19.812 INFO:teuthology.orchestra.run.smithi196.stdout: auth debug = true 2020-06-04T08:24:19.812 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.812 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on old message = true 2020-06-04T08:24:19.812 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on bug = true 2020-06-04T08:24:19.812 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.813 INFO:teuthology.orchestra.run.smithi196.stdout: mon max pg per osd = 10000 # >= luminous 2020-06-04T08:24:19.813 INFO:teuthology.orchestra.run.smithi196.stdout: mon pg warn max object skew = 0 2020-06-04T08:24:19.813 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.813 INFO:teuthology.orchestra.run.smithi196.stdout: # disable pg_autoscaler by default for new pools 2020-06-04T08:24:19.813 INFO:teuthology.orchestra.run.smithi196.stdout: osd_pool_default_pg_autoscale_mode = off 2020-06-04T08:24:19.814 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.814 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default size = 2 2020-06-04T08:24:19.814 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.814 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow primary affinity = true 2020-06-04T08:24:19.814 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow pg remap = true 2020-06-04T08:24:19.814 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on legacy crush tunables = false 2020-06-04T08:24:19.815 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on crush straw calc version zero = false 2020-06-04T08:24:19.815 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on no sortbitwise = false 2020-06-04T08:24:19.815 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on osd down out interval zero = false 2020-06-04T08:24:19.815 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on too few osds = false 2020-06-04T08:24:19.815 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-06-04T08:24:19.816 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_no_redundancy = false 2020-06-04T08:24:19.816 INFO:teuthology.orchestra.run.smithi196.stdout: mon_allow_pool_size_one = true 2020-06-04T08:24:19.816 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.816 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-06-04T08:24:19.816 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.817 INFO:teuthology.orchestra.run.smithi196.stdout: osd default data pool replay window = 5 2020-06-04T08:24:19.817 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.817 INFO:teuthology.orchestra.run.smithi196.stdout: mon allow pool delete = true 2020-06-04T08:24:19.817 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.817 INFO:teuthology.orchestra.run.smithi196.stdout: mon cluster log file level = debug 2020-06-04T08:24:19.818 INFO:teuthology.orchestra.run.smithi196.stdout: debug asserts on shutdown = true 2020-06-04T08:24:19.818 INFO:teuthology.orchestra.run.smithi196.stdout: mon host = 172.21.15.60 2020-06-04T08:24:19.818 INFO:teuthology.orchestra.run.smithi196.stdout: osd heartbeat grace = 40 2020-06-04T08:24:19.818 INFO:teuthology.orchestra.run.smithi196.stdout: fsid = 959b9e79-8d6d-4cc9-b316-d7687f41bb88 2020-06-04T08:24:19.818 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.819 INFO:teuthology.orchestra.run.smithi196.stdout:[osd] 2020-06-04T08:24:19.819 INFO:teuthology.orchestra.run.smithi196.stdout: osd journal size = 100 2020-06-04T08:24:19.819 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.819 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub load threshold = 5.0 2020-06-04T08:24:19.819 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub max interval = 600 2020-06-04T08:24:19.820 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.820 INFO:teuthology.orchestra.run.smithi196.stdout: osd recover clone overlap = true 2020-06-04T08:24:19.820 INFO:teuthology.orchestra.run.smithi196.stdout: osd recovery max chunk = 1048576 2020-06-04T08:24:19.820 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.820 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug shutdown = true 2020-06-04T08:24:19.820 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug op order = true 2020-06-04T08:24:19.821 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug verify stray on activate = true 2020-06-04T08:24:19.821 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.821 INFO:teuthology.orchestra.run.smithi196.stdout: osd open classes on start = true 2020-06-04T08:24:19.821 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug pg log writeout = true 2020-06-04T08:24:19.821 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.822 INFO:teuthology.orchestra.run.smithi196.stdout: osd deep scrub update digest min age = 30 2020-06-04T08:24:19.822 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.822 INFO:teuthology.orchestra.run.smithi196.stdout: osd map max advance = 10 2020-06-04T08:24:19.822 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.822 INFO:teuthology.orchestra.run.smithi196.stdout: journal zero on create = true 2020-06-04T08:24:19.823 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.823 INFO:teuthology.orchestra.run.smithi196.stdout: filestore ondisk finisher threads = 3 2020-06-04T08:24:19.823 INFO:teuthology.orchestra.run.smithi196.stdout: filestore apply finisher threads = 3 2020-06-04T08:24:19.823 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.823 INFO:teuthology.orchestra.run.smithi196.stdout: bdev debug aio = true 2020-06-04T08:24:19.823 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug misdirected ops = true 2020-06-04T08:24:19.824 INFO:teuthology.orchestra.run.smithi196.stdout: debug filestore = 20 2020-06-04T08:24:19.824 INFO:teuthology.orchestra.run.smithi196.stdout: debug journal = 20 2020-06-04T08:24:19.824 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 20 2020-06-04T08:24:19.824 INFO:teuthology.orchestra.run.smithi196.stdout: debug osd = 25 2020-06-04T08:24:19.824 INFO:teuthology.orchestra.run.smithi196.stdout: osd fast shutdown = False 2020-06-04T08:24:19.825 INFO:teuthology.orchestra.run.smithi196.stdout: rocksdb delete range threshold = 0 2020-06-04T08:24:19.825 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.825 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr] 2020-06-04T08:24:19.825 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2020-06-04T08:24:19.825 INFO:teuthology.orchestra.run.smithi196.stdout: debug mgr = 20 2020-06-04T08:24:19.825 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2020-06-04T08:24:19.826 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2020-06-04T08:24:19.826 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2020-06-04T08:24:19.826 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2020-06-04T08:24:19.826 INFO:teuthology.orchestra.run.smithi196.stdout: mgr/telemetry/nag = false 2020-06-04T08:24:19.826 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.827 INFO:teuthology.orchestra.run.smithi196.stdout:[mon] 2020-06-04T08:24:19.827 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2020-06-04T08:24:19.827 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2020-06-04T08:24:19.827 INFO:teuthology.orchestra.run.smithi196.stdout: debug paxos = 20 2020-06-04T08:24:19.827 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2020-06-04T08:24:19.827 INFO:teuthology.orchestra.run.smithi196.stdout: mon data avail warn = 5 2020-06-04T08:24:19.828 INFO:teuthology.orchestra.run.smithi196.stdout: mon mgr mkfs grace = 240 2020-06-04T08:24:19.828 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2020-06-04T08:24:19.828 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd reporter subtree level = osd 2020-06-04T08:24:19.828 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd prime pg temp = true 2020-06-04T08:24:19.828 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2020-06-04T08:24:19.829 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd crush smoke test = False 2020-06-04T08:24:19.829 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:19.829 INFO:teuthology.orchestra.run.smithi196.stdout:[client] 2020-06-04T08:24:19.829 INFO:teuthology.orchestra.run.smithi196.stdout: rgw cache enabled = true 2020-06-04T08:24:19.829 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable ops log = true 2020-06-04T08:24:19.829 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable usage log = true 2020-06-04T08:24:19.830 INFO:teuthology.orchestra.run.smithi196.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-06-04T08:24:19.830 INFO:teuthology.orchestra.run.smithi196.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-06-04T08:24:19.830 INFO:teuthology.orchestra.run.smithi196.stdout: debug rgw = 20 2020-06-04T08:24:19.830 INFO:teuthology.orchestra.run.smithi196.stdout: rgw crypt require ssl = False 2020-06-04T08:24:19.830 INFO:teuthology.orchestra.run.smithi196.stdout: rgw crypt s3 kms backend = testing 2020-06-04T08:24:19.831 INFO:teuthology.orchestra.run.smithi196.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-06-04T08:24:19.831 INFO:teuthology.orchestra.run.smithi196.stdout: rgw curl low speed time = 300 2020-06-04T08:24:19.831 INFO:teuthology.orchestra.run.smithi196.stdout: rgw data log num shards = 4 2020-06-04T08:24:19.831 INFO:teuthology.orchestra.run.smithi196.stdout: rgw md log max shards = 4 2020-06-04T08:24:19.831 INFO:teuthology.orchestra.run.smithi196.stdout: rgw sync log trim interval = 0 2020-06-04T08:24:19.831 INFO:teuthology.orchestra.run.smithi196.stdout: setgroup = ceph 2020-06-04T08:24:19.832 INFO:teuthology.orchestra.run.smithi196.stdout: setuser = ceph 2020-06-04T08:24:19.832 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.a] 2020-06-04T08:24:19.832 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.0] 2020-06-04T08:24:19.832 INFO:teuthology.orchestra.run.smithi196.stdout: osd_max_omap_entries_per_request = 10 2020-06-04T08:24:19.832 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.1] 2020-06-04T08:24:19.833 INFO:teuthology.orchestra.run.smithi196.stdout: osd_max_omap_entries_per_request = 1000 2020-06-04T08:24:19.833 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.2] 2020-06-04T08:24:19.833 INFO:teuthology.orchestra.run.smithi196.stdout: osd_max_omap_entries_per_request = 10000 2020-06-04T08:24:19.833 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2020-06-04T08:24:19.834 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2020-06-04T08:24:19.905 INFO:tasks.ceph:Copying monmap to all nodes... 2020-06-04T08:24:19.954 DEBUG:teuthology.orchestra.remote:smithi060:/etc/ceph/c1.keyring is 206B 2020-06-04T08:24:19.972 DEBUG:teuthology.orchestra.remote:smithi060:/home/ubuntu/cephtest/c1.monmap is 179B 2020-06-04T08:24:19.982 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi060.front.sepia.ceph.com 2020-06-04T08:24:19.983 INFO:teuthology.orchestra.run.smithi060:> sudo sh -c 'cat > /etc/ceph/c1.keyring' && sudo chmod 0644 /etc/ceph/c1.keyring 2020-06-04T08:24:20.038 INFO:teuthology.orchestra.run.smithi060:> cat > /home/ubuntu/cephtest/c1.monmap 2020-06-04T08:24:20.127 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi196.front.sepia.ceph.com 2020-06-04T08:24:20.128 INFO:teuthology.orchestra.run.smithi196:> sudo sh -c 'cat > /etc/ceph/c1.keyring' && sudo chmod 0644 /etc/ceph/c1.keyring 2020-06-04T08:24:20.187 INFO:teuthology.orchestra.run.smithi196:> cat > /home/ubuntu/cephtest/c1.monmap 2020-06-04T08:24:20.277 INFO:tasks.ceph:Setting up mon nodes... 2020-06-04T08:24:20.277 INFO:tasks.ceph:Setting up mgr nodes... 2020-06-04T08:24:20.278 INFO:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2020-06-04T08:24:20.320 INFO:teuthology.orchestra.run.smithi060.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2020-06-04T08:24:20.323 INFO:tasks.ceph:Setting up mds nodes... 2020-06-04T08:24:20.324 INFO:tasks.ceph_client:Setting up client nodes... 2020-06-04T08:24:20.325 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2020-06-04T08:24:20.399 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/c1.client.0.keyring 2020-06-04T08:24:20.410 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2020-06-04T08:24:20.492 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/c1.client.1.keyring 2020-06-04T08:24:20.499 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-06-04T08:24:20.499 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2020-06-04T08:24:20.500 INFO:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2020-06-04T08:24:20.556 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-06-04T08:24:20.557 INFO:tasks.ceph:role: c1.osd.0 2020-06-04T08:24:20.557 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com 2020-06-04T08:24:20.557 INFO:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-06-04T08:24:20.640 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-06-04T08:24:20.641 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2020-06-04T08:24:20.641 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-06-04T08:24:20.641 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-06-04T08:24:20.642 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2020-06-04T08:24:20.642 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-06-04T08:24:20.642 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-06-04T08:24:20.642 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-06-04T08:24:20.643 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-06-04T08:24:20.644 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2020-06-04T08:24:20.644 INFO:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2020-06-04T08:24:20.707 INFO:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2020-06-04T08:24:20.761 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: /sbin/restorecon: command not found 2020-06-04T08:24:20.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-06-04T08:24:20.763 INFO:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2020-06-04T08:24:20.828 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-06-04T08:24:20.829 INFO:tasks.ceph:role: c1.osd.1 2020-06-04T08:24:20.829 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com 2020-06-04T08:24:20.830 INFO:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-06-04T08:24:20.913 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-06-04T08:24:20.914 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2020-06-04T08:24:20.914 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-06-04T08:24:20.914 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-06-04T08:24:20.914 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2020-06-04T08:24:20.914 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-06-04T08:24:20.915 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-06-04T08:24:20.915 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-06-04T08:24:20.915 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-06-04T08:24:20.916 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2020-06-04T08:24:20.916 INFO:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2020-06-04T08:24:20.983 INFO:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2020-06-04T08:24:21.038 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: /sbin/restorecon: command not found 2020-06-04T08:24:21.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-06-04T08:24:21.040 INFO:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2020-06-04T08:24:21.097 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2020-06-04T08:24:21.097 INFO:tasks.ceph:role: c1.osd.2 2020-06-04T08:24:21.098 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com 2020-06-04T08:24:21.105 INFO:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-06-04T08:24:21.181 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-06-04T08:24:21.181 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2020-06-04T08:24:21.182 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-06-04T08:24:21.182 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-06-04T08:24:21.182 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2020-06-04T08:24:21.182 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-06-04T08:24:21.183 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-06-04T08:24:21.183 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-06-04T08:24:21.183 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-06-04T08:24:21.184 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2020-06-04T08:24:21.184 INFO:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2020-06-04T08:24:21.246 INFO:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2020-06-04T08:24:21.300 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: /sbin/restorecon: command not found 2020-06-04T08:24:21.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-06-04T08:24:21.301 INFO:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2020-06-04T08:24:21.401 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:24:21.395+0000 7f869e137d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2020-06-04T08:24:21.402 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:24:21.395+0000 7f869e137d80 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2020-06-04T08:24:21.402 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:24:21.395+0000 7f869e137d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-06-04T08:24:21.402 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:24:21.395+0000 7f869e137d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-06-04T08:24:21.403 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:24:21.395+0000 7f869e137d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2020-06-04T08:24:21.403 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:24:21.399+0000 7f869e137d80 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2020-06-04T08:24:23.322 INFO:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2020-06-04T08:24:23.339 INFO:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2020-06-04T08:24:23.424 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:24:23.419+0000 7ff611ff3d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2020-06-04T08:24:23.424 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:24:23.419+0000 7ff611ff3d80 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2020-06-04T08:24:23.425 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:24:23.419+0000 7ff611ff3d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-06-04T08:24:23.425 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:24:23.419+0000 7ff611ff3d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-06-04T08:24:23.425 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:24:23.419+0000 7ff611ff3d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2020-06-04T08:24:23.426 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:24:23.419+0000 7ff611ff3d80 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2020-06-04T08:24:25.362 INFO:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2020-06-04T08:24:25.422 INFO:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2020-06-04T08:24:25.509 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:24:25.503+0000 7f1da0b24d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2020-06-04T08:24:25.510 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:24:25.503+0000 7f1da0b24d80 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2020-06-04T08:24:25.510 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:24:25.503+0000 7f1da0b24d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-06-04T08:24:25.510 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:24:25.503+0000 7f1da0b24d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-06-04T08:24:25.511 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:24:25.503+0000 7f1da0b24d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2020-06-04T08:24:25.512 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:24:25.503+0000 7f1da0b24d80 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2020-06-04T08:24:27.429 INFO:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2020-06-04T08:24:27.485 INFO:tasks.ceph:Reading keys from all nodes... 2020-06-04T08:24:27.485 INFO:teuthology.orchestra.run.smithi060:> mktemp 2020-06-04T08:24:27.531 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.JFqsEem5Rz 2020-06-04T08:24:27.531 INFO:teuthology.orchestra.run.smithi060:> sudo cp /var/lib/ceph/mgr/c1-x/keyring /tmp/tmp.JFqsEem5Rz 2020-06-04T08:24:27.588 INFO:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.JFqsEem5Rz 2020-06-04T08:24:27.693 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.JFqsEem5Rz is 56B 2020-06-04T08:24:27.704 INFO:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.JFqsEem5Rz 2020-06-04T08:24:27.711 INFO:teuthology.orchestra.run.smithi060:> mktemp 2020-06-04T08:24:27.759 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.LSHfVimK5P 2020-06-04T08:24:27.760 INFO:teuthology.orchestra.run.smithi060:> sudo cp /var/lib/ceph/osd/c1-0/keyring /tmp/tmp.LSHfVimK5P 2020-06-04T08:24:27.817 INFO:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.LSHfVimK5P 2020-06-04T08:24:27.922 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.LSHfVimK5P is 56B 2020-06-04T08:24:27.933 INFO:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.LSHfVimK5P 2020-06-04T08:24:27.939 INFO:teuthology.orchestra.run.smithi060:> mktemp 2020-06-04T08:24:27.987 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.It8OTnHdpq 2020-06-04T08:24:27.988 INFO:teuthology.orchestra.run.smithi060:> sudo cp /var/lib/ceph/osd/c1-1/keyring /tmp/tmp.It8OTnHdpq 2020-06-04T08:24:28.045 INFO:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.It8OTnHdpq 2020-06-04T08:24:28.153 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.It8OTnHdpq is 56B 2020-06-04T08:24:28.164 INFO:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.It8OTnHdpq 2020-06-04T08:24:28.170 INFO:teuthology.orchestra.run.smithi060:> mktemp 2020-06-04T08:24:28.219 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.oaQuSz739g 2020-06-04T08:24:28.220 INFO:teuthology.orchestra.run.smithi060:> sudo cp /var/lib/ceph/osd/c1-2/keyring /tmp/tmp.oaQuSz739g 2020-06-04T08:24:28.282 INFO:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.oaQuSz739g 2020-06-04T08:24:28.389 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.oaQuSz739g is 56B 2020-06-04T08:24:28.438 INFO:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.oaQuSz739g 2020-06-04T08:24:28.497 DEBUG:teuthology.orchestra.remote:smithi060:/etc/ceph/c1.client.0.keyring is 59B 2020-06-04T08:24:28.556 DEBUG:teuthology.orchestra.remote:smithi060:/etc/ceph/c1.client.1.keyring is 59B 2020-06-04T08:24:28.611 INFO:tasks.ceph:Adding keys to all mons... 2020-06-04T08:24:28.613 INFO:teuthology.orchestra.run.smithi060:> sudo tee -a /etc/ceph/c1.keyring 2020-06-04T08:24:28.660 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr.x] 2020-06-04T08:24:28.661 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQC0r9hepz4QExAA52EhWSp88M6ir0OOedF2yA== 2020-06-04T08:24:28.661 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.0] 2020-06-04T08:24:28.661 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQC1r9he3QvdFxAAnwwZQPDBj+Z+AgE/AxJVFw== 2020-06-04T08:24:28.661 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.1] 2020-06-04T08:24:28.661 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQC3r9hebFQyGRAA3ivjGheKcpgAvEVjeSpqiQ== 2020-06-04T08:24:28.662 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.2] 2020-06-04T08:24:28.662 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQC5r9heLhtGHhAAiZcmeieKUlWRmrf/HCHPkQ== 2020-06-04T08:24:28.671 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2020-06-04T08:24:28.671 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQC0r9heFmm3FxAAWXjIEAXLp0+keYcqn5G/9w== 2020-06-04T08:24:28.671 INFO:teuthology.orchestra.run.smithi060.stdout:[client.1] 2020-06-04T08:24:28.672 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQC0r9heOXX8HBAA7zwfQjTWBeob8xqkc7UaVQ== 2020-06-04T08:24:28.672 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-06-04T08:24:28.744 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-04T08:24:28.830 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-04T08:24:28.926 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-04T08:24:29.011 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-06-04T08:24:29.099 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-06-04T08:24:29.144 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-06-04T08:24:29.145 INFO:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2020-06-04T08:24:29.199 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2020-06-04T08:24:29.295 INFO:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2020-06-04T08:24:29.310 INFO:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/c1.monmap 2020-06-04T08:24:29.359 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2020-06-04T08:24:29.360 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2020-06-04T08:24:29.361 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2020-06-04T08:24:29.403 INFO:tasks.ceph.c1.mon.a:Started 2020-06-04T08:24:29.404 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2020-06-04T08:24:29.404 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2020-06-04T08:24:29.405 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2020-06-04T08:24:29.408 INFO:tasks.ceph.c1.mgr.x:Started 2020-06-04T08:24:29.409 INFO:tasks.ceph:Setting crush tunables to default 2020-06-04T08:24:29.410 INFO:teuthology.orchestra.run.smithi060:> sudo ceph --cluster c1 osd crush tunables default 2020-06-04T08:24:29.738 INFO:teuthology.orchestra.run.smithi060.stderr:adjusted tunables profile to default 2020-06-04T08:24:29.749 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2020-06-04T08:24:29.749 INFO:teuthology.orchestra.run.smithi060:> mktemp 2020-06-04T08:24:29.754 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.7gnjhgoCRX 2020-06-04T08:24:29.754 INFO:teuthology.orchestra.run.smithi060:> sudo cp /var/lib/ceph/osd/c1-0/fsid /tmp/tmp.7gnjhgoCRX 2020-06-04T08:24:29.808 INFO:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.7gnjhgoCRX 2020-06-04T08:24:29.914 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.7gnjhgoCRX is 37B 2020-06-04T08:24:29.968 INFO:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.7gnjhgoCRX 2020-06-04T08:24:29.976 INFO:teuthology.orchestra.run.smithi060:> mktemp 2020-06-04T08:24:30.023 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.fjKGQpvLVG 2020-06-04T08:24:30.024 INFO:teuthology.orchestra.run.smithi060:> sudo cp /var/lib/ceph/osd/c1-1/fsid /tmp/tmp.fjKGQpvLVG 2020-06-04T08:24:30.083 INFO:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.fjKGQpvLVG 2020-06-04T08:24:30.191 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.fjKGQpvLVG is 37B 2020-06-04T08:24:30.244 INFO:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.fjKGQpvLVG 2020-06-04T08:24:30.253 INFO:teuthology.orchestra.run.smithi060:> mktemp 2020-06-04T08:24:30.307 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.Ea9rcZmSAG 2020-06-04T08:24:30.307 INFO:teuthology.orchestra.run.smithi060:> sudo cp /var/lib/ceph/osd/c1-2/fsid /tmp/tmp.Ea9rcZmSAG 2020-06-04T08:24:30.365 INFO:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.Ea9rcZmSAG 2020-06-04T08:24:30.470 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.Ea9rcZmSAG is 37B 2020-06-04T08:24:30.537 INFO:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.Ea9rcZmSAG 2020-06-04T08:24:30.544 INFO:teuthology.orchestra.run.smithi060:> sudo ceph --cluster c1 osd new b1fab183-7f7d-4d5d-831d-b48f5130f20b 0 2020-06-04T08:24:30.787 INFO:teuthology.orchestra.run.smithi060.stdout:0 2020-06-04T08:24:30.800 INFO:teuthology.orchestra.run.smithi060:> sudo ceph --cluster c1 osd new ec4787e0-dcc8-4ffa-9bff-394e7837730b 1 2020-06-04T08:24:31.000 INFO:teuthology.orchestra.run.smithi060.stdout:1 2020-06-04T08:24:31.016 INFO:teuthology.orchestra.run.smithi060:> sudo ceph --cluster c1 osd new c9476f9a-3da5-4adc-a36f-41e837984382 2 2020-06-04T08:24:31.231 INFO:teuthology.orchestra.run.smithi060.stdout:2 2020-06-04T08:24:31.232 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2020-06-04T08:24:31.233 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2020-06-04T08:24:31.236 INFO:tasks.ceph.c1.osd.0:Started 2020-06-04T08:24:31.237 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2020-06-04T08:24:31.237 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2020-06-04T08:24:31.255 INFO:tasks.ceph.c1.osd.1:Started 2020-06-04T08:24:31.255 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2020-06-04T08:24:31.256 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2020-06-04T08:24:31.265 INFO:tasks.ceph.c1.osd.2:Started 2020-06-04T08:24:31.266 INFO:tasks.ceph:Waiting for OSDs to come up 2020-06-04T08:24:31.569 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-06-04T08:24:31.802 INFO:teuthology.misc.health.smithi060.stdout: 2020-06-04T08:24:31.803 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":5,"fsid":"959b9e79-8d6d-4cc9-b316-d7687f41bb88","created":"2020-06-04T08:24:29.525298+0000","modified":"2020-06-04T08:24:31.213828+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"b1fab183-7f7d-4d5d-831d-b48f5130f20b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ec4787e0-dcc8-4ffa-9bff-394e7837730b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c9476f9a-3da5-4adc-a36f-41e837984382","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-04T08:24:31.812 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-06-04T08:24:31.856 INFO:tasks.ceph.c1.osd.2.smithi060.stderr:2020-06-04T08:24:31.851+0000 7f22f57b1d80 -1 Falling back to public interface 2020-06-04T08:24:31.864 INFO:tasks.ceph.c1.osd.0.smithi060.stderr:2020-06-04T08:24:31.859+0000 7fe7c87a6d80 -1 Falling back to public interface 2020-06-04T08:24:31.865 INFO:tasks.ceph.c1.osd.1.smithi060.stderr:2020-06-04T08:24:31.859+0000 7f416a1c3d80 -1 Falling back to public interface 2020-06-04T08:24:32.453 INFO:tasks.ceph.c1.osd.2.smithi060.stderr:2020-06-04T08:24:32.447+0000 7f22f57b1d80 -1 osd.2 0 log_to_monitors {default=true} 2020-06-04T08:24:32.482 INFO:tasks.ceph.c1.osd.1.smithi060.stderr:2020-06-04T08:24:32.479+0000 7f416a1c3d80 -1 osd.1 0 log_to_monitors {default=true} 2020-06-04T08:24:32.489 INFO:tasks.ceph.c1.osd.0.smithi060.stderr:2020-06-04T08:24:32.483+0000 7fe7c87a6d80 -1 osd.0 0 log_to_monitors {default=true} 2020-06-04T08:24:33.557 INFO:tasks.ceph.c1.osd.2.smithi060.stderr:2020-06-04T08:24:33.551+0000 7f22ecc7c700 -1 osd.2 0 waiting for initial osdmap 2020-06-04T08:24:33.558 INFO:tasks.ceph.c1.osd.0.smithi060.stderr:2020-06-04T08:24:33.551+0000 7fe7bfc71700 -1 osd.0 0 waiting for initial osdmap 2020-06-04T08:24:33.558 INFO:tasks.ceph.c1.osd.1.smithi060.stderr:2020-06-04T08:24:33.555+0000 7f416168e700 -1 osd.1 0 waiting for initial osdmap 2020-06-04T08:24:38.116 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2020-06-04T08:24:38.388 INFO:teuthology.misc.health.smithi060.stdout: 2020-06-04T08:24:38.390 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"959b9e79-8d6d-4cc9-b316-d7687f41bb88","created":"2020-06-04T08:24:29.525298+0000","modified":"2020-06-04T08:24:37.411747+0000","last_up_change":"2020-06-04T08:24:34.548310+0000","last_in_change":"2020-06-04T08:24:34.548310+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-04T08:24:33.421703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"b1fab183-7f7d-4d5d-831d-b48f5130f20b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6809","nonce":13672},{"type":"v1","addr":"172.21.15.60:6811","nonce":13672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":13672},{"type":"v1","addr":"172.21.15.60:6815","nonce":13672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":13672},{"type":"v1","addr":"172.21.15.60:6823","nonce":13672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":13672},{"type":"v1","addr":"172.21.15.60:6820","nonce":13672}]},"public_addr":"172.21.15.60:6811/13672","cluster_addr":"172.21.15.60:6815/13672","heartbeat_back_addr":"172.21.15.60:6823/13672","heartbeat_front_addr":"172.21.15.60:6820/13672","state":["exists","up"]},{"osd":1,"uuid":"ec4787e0-dcc8-4ffa-9bff-394e7837730b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":13674},{"type":"v1","addr":"172.21.15.60:6810","nonce":13674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":13674},{"type":"v1","addr":"172.21.15.60:6814","nonce":13674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":13674},{"type":"v1","addr":"172.21.15.60:6821","nonce":13674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":13674},{"type":"v1","addr":"172.21.15.60:6817","nonce":13674}]},"public_addr":"172.21.15.60:6810/13674","cluster_addr":"172.21.15.60:6814/13674","heartbeat_back_addr":"172.21.15.60:6821/13674","heartbeat_front_addr":"172.21.15.60:6817/13674","state":["exists","up"]},{"osd":2,"uuid":"c9476f9a-3da5-4adc-a36f-41e837984382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":13673},{"type":"v1","addr":"172.21.15.60:6801","nonce":13673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":13673},{"type":"v1","addr":"172.21.15.60:6803","nonce":13673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":13673},{"type":"v1","addr":"172.21.15.60:6807","nonce":13673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":13673},{"type":"v1","addr":"172.21.15.60:6805","nonce":13673}]},"public_addr":"172.21.15.60:6801/13673","cluster_addr":"172.21.15.60:6803/13673","heartbeat_back_addr":"172.21.15.60:6807/13673","heartbeat_front_addr":"172.21.15.60:6805/13673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:24:33.460710+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:24:33.485977+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:24:33.486162+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-04T08:24:38.400 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-06-04T08:24:38.401 INFO:tasks.ceph:Creating RBD pool 2020-06-04T08:24:38.401 INFO:teuthology.orchestra.run.smithi060:> sudo ceph --cluster c1 osd pool create rbd 8 2020-06-04T08:24:39.693 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'rbd' created 2020-06-04T08:24:39.706 INFO:teuthology.orchestra.run.smithi060:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-06-04T08:24:40.700 INFO:teuthology.orchestra.run.smithi060.stderr:enabled application 'rbd' on pool 'rbd' 2020-06-04T08:24:40.714 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2020-06-04T08:24:40.715 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-06-04T08:24:40.716 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-06-04T08:24:40.978 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:40.978 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":12,"fsid":"959b9e79-8d6d-4cc9-b316-d7687f41bb88","created":"2020-06-04T08:24:29.525298+0000","modified":"2020-06-04T08:24:40.679459+0000","last_up_change":"2020-06-04T08:24:34.548310+0000","last_in_change":"2020-06-04T08:24:34.548310+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-04T08:24:33.421703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-06-04T08:24:38.677282+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b1fab183-7f7d-4d5d-831d-b48f5130f20b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6809","nonce":13672},{"type":"v1","addr":"172.21.15.60:6811","nonce":13672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":13672},{"type":"v1","addr":"172.21.15.60:6815","nonce":13672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":13672},{"type":"v1","addr":"172.21.15.60:6823","nonce":13672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":13672},{"type":"v1","addr":"172.21.15.60:6820","nonce":13672}]},"public_addr":"172.21.15.60:6811/13672","cluster_addr":"172.21.15.60:6815/13672","heartbeat_back_addr":"172.21.15.60:6823/13672","heartbeat_front_addr":"172.21.15.60:6820/13672","state":["exists","up"]},{"osd":1,"uuid":"ec4787e0-dcc8-4ffa-9bff-394e7837730b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":13674},{"type":"v1","addr":"172.21.15.60:6810","nonce":13674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":13674},{"type":"v1","addr":"172.21.15.60:6814","nonce":13674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":13674},{"type":"v1","addr":"172.21.15.60:6821","nonce":13674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":13674},{"type":"v1","addr":"172.21.15.60:6817","nonce":13674}]},"public_addr":"172.21.15.60:6810/13674","cluster_addr":"172.21.15.60:6814/13674","heartbeat_back_addr":"172.21.15.60:6821/13674","heartbeat_front_addr":"172.21.15.60:6817/13674","state":["exists","up"]},{"osd":2,"uuid":"c9476f9a-3da5-4adc-a36f-41e837984382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":13673},{"type":"v1","addr":"172.21.15.60:6801","nonce":13673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":13673},{"type":"v1","addr":"172.21.15.60:6803","nonce":13673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":13673},{"type":"v1","addr":"172.21.15.60:6807","nonce":13673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":13673},{"type":"v1","addr":"172.21.15.60:6805","nonce":13673}]},"public_addr":"172.21.15.60:6801/13673","cluster_addr":"172.21.15.60:6803/13673","heartbeat_back_addr":"172.21.15.60:6807/13673","heartbeat_front_addr":"172.21.15.60:6805/13673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:24:33.460710+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:24:33.485977+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:24:33.486162+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-04T08:24:40.990 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-06-04T08:24:33.421703+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-06-04T08:24:38.677282+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-06-04T08:24:40.991 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get device_health_metrics pg_num 2020-06-04T08:24:41.261 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 1 2020-06-04T08:24:41.275 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get rbd pg_num 2020-06-04T08:24:41.551 INFO:teuthology.orchestra.run.smithi060.stdout:pg_num: 8 2020-06-04T08:24:41.556 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2020-06-04T08:24:41.556 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2020-06-04T08:24:41.557 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early mgr dump --format=json 2020-06-04T08:24:41.866 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:41.881 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":4,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6824","nonce":13471},{"type":"v1","addr":"172.21.15.60:6825","nonce":13471}]},"active_addr":"172.21.15.60:6825/13471","active_change":"2020-06-04T08:24:33.390948+0000","active_mgr_features":4540138292837744639,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"ceph/ceph-grafana:latest","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","osd_support","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","osd_support","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":3402805930}]},{"addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":3823577783}]},{"addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":783468214}]}]}} 2020-06-04T08:24:41.886 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2020-06-04T08:24:41.886 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2020-06-04T08:24:41.887 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-06-04T08:24:42.152 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:42.153 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":13,"fsid":"959b9e79-8d6d-4cc9-b316-d7687f41bb88","created":"2020-06-04T08:24:29.525298+0000","modified":"2020-06-04T08:24:41.685552+0000","last_up_change":"2020-06-04T08:24:34.548310+0000","last_in_change":"2020-06-04T08:24:34.548310+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-04T08:24:33.421703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-06-04T08:24:38.677282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b1fab183-7f7d-4d5d-831d-b48f5130f20b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6809","nonce":13672},{"type":"v1","addr":"172.21.15.60:6811","nonce":13672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":13672},{"type":"v1","addr":"172.21.15.60:6815","nonce":13672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":13672},{"type":"v1","addr":"172.21.15.60:6823","nonce":13672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":13672},{"type":"v1","addr":"172.21.15.60:6820","nonce":13672}]},"public_addr":"172.21.15.60:6811/13672","cluster_addr":"172.21.15.60:6815/13672","heartbeat_back_addr":"172.21.15.60:6823/13672","heartbeat_front_addr":"172.21.15.60:6820/13672","state":["exists","up"]},{"osd":1,"uuid":"ec4787e0-dcc8-4ffa-9bff-394e7837730b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":13674},{"type":"v1","addr":"172.21.15.60:6810","nonce":13674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":13674},{"type":"v1","addr":"172.21.15.60:6814","nonce":13674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":13674},{"type":"v1","addr":"172.21.15.60:6821","nonce":13674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":13674},{"type":"v1","addr":"172.21.15.60:6817","nonce":13674}]},"public_addr":"172.21.15.60:6810/13674","cluster_addr":"172.21.15.60:6814/13674","heartbeat_back_addr":"172.21.15.60:6821/13674","heartbeat_front_addr":"172.21.15.60:6817/13674","state":["exists","up"]},{"osd":2,"uuid":"c9476f9a-3da5-4adc-a36f-41e837984382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":13673},{"type":"v1","addr":"172.21.15.60:6801","nonce":13673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":13673},{"type":"v1","addr":"172.21.15.60:6803","nonce":13673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":13673},{"type":"v1","addr":"172.21.15.60:6807","nonce":13673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":13673},{"type":"v1","addr":"172.21.15.60:6805","nonce":13673}]},"public_addr":"172.21.15.60:6801/13673","cluster_addr":"172.21.15.60:6803/13673","heartbeat_back_addr":"172.21.15.60:6807/13673","heartbeat_front_addr":"172.21.15.60:6805/13673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:24:33.460710+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:24:33.485977+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:24:33.486162+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-04T08:24:42.167 INFO:tasks.ceph.ceph_manager.c1:all up! 2020-06-04T08:24:42.168 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2020-06-04T08:24:42.433 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:42.433 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":13,"fsid":"959b9e79-8d6d-4cc9-b316-d7687f41bb88","created":"2020-06-04T08:24:29.525298+0000","modified":"2020-06-04T08:24:41.685552+0000","last_up_change":"2020-06-04T08:24:34.548310+0000","last_in_change":"2020-06-04T08:24:34.548310+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-04T08:24:33.421703+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-06-04T08:24:38.677282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"b1fab183-7f7d-4d5d-831d-b48f5130f20b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6809","nonce":13672},{"type":"v1","addr":"172.21.15.60:6811","nonce":13672}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6813","nonce":13672},{"type":"v1","addr":"172.21.15.60:6815","nonce":13672}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6822","nonce":13672},{"type":"v1","addr":"172.21.15.60:6823","nonce":13672}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6818","nonce":13672},{"type":"v1","addr":"172.21.15.60:6820","nonce":13672}]},"public_addr":"172.21.15.60:6811/13672","cluster_addr":"172.21.15.60:6815/13672","heartbeat_back_addr":"172.21.15.60:6823/13672","heartbeat_front_addr":"172.21.15.60:6820/13672","state":["exists","up"]},{"osd":1,"uuid":"ec4787e0-dcc8-4ffa-9bff-394e7837730b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":13674},{"type":"v1","addr":"172.21.15.60:6810","nonce":13674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6812","nonce":13674},{"type":"v1","addr":"172.21.15.60:6814","nonce":13674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6819","nonce":13674},{"type":"v1","addr":"172.21.15.60:6821","nonce":13674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6816","nonce":13674},{"type":"v1","addr":"172.21.15.60:6817","nonce":13674}]},"public_addr":"172.21.15.60:6810/13674","cluster_addr":"172.21.15.60:6814/13674","heartbeat_back_addr":"172.21.15.60:6821/13674","heartbeat_front_addr":"172.21.15.60:6817/13674","state":["exists","up"]},{"osd":2,"uuid":"c9476f9a-3da5-4adc-a36f-41e837984382","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":13673},{"type":"v1","addr":"172.21.15.60:6801","nonce":13673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":13673},{"type":"v1","addr":"172.21.15.60:6803","nonce":13673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":13673},{"type":"v1","addr":"172.21.15.60:6807","nonce":13673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":13673},{"type":"v1","addr":"172.21.15.60:6805","nonce":13673}]},"public_addr":"172.21.15.60:6801/13673","cluster_addr":"172.21.15.60:6803/13673","heartbeat_back_addr":"172.21.15.60:6807/13673","heartbeat_front_addr":"172.21.15.60:6805/13673","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:24:33.460710+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:24:33.485977+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:24:33.486162+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-04T08:24:42.455 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.0 flush_pg_stats 2020-06-04T08:24:42.615 INFO:teuthology.orchestra.run.smithi060.stdout:34359738372 2020-06-04T08:24:42.616 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.1 flush_pg_stats 2020-06-04T08:24:42.777 INFO:teuthology.orchestra.run.smithi060.stdout:34359738372 2020-06-04T08:24:42.778 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.2 flush_pg_stats 2020-06-04T08:24:42.942 INFO:teuthology.orchestra.run.smithi060.stdout:34359738371 2020-06-04T08:24:42.943 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2020-06-04T08:24:43.211 INFO:teuthology.orchestra.run.smithi060.stdout:34359738371 2020-06-04T08:24:43.225 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738371 for osd.0 2020-06-04T08:24:44.226 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2020-06-04T08:24:44.507 INFO:teuthology.orchestra.run.smithi060.stdout:34359738372 2020-06-04T08:24:44.520 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.0 2020-06-04T08:24:44.521 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.1 2020-06-04T08:24:44.784 INFO:teuthology.orchestra.run.smithi060.stdout:34359738372 2020-06-04T08:24:44.803 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.1 2020-06-04T08:24:44.804 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.2 2020-06-04T08:24:45.076 INFO:teuthology.orchestra.run.smithi060.stdout:34359738371 2020-06-04T08:24:45.089 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.2 2020-06-04T08:24:45.089 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2020-06-04T08:24:45.090 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2020-06-04T08:24:45.351 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:45.353 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2020-06-04T08:24:45.369 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2020-06-04T08:24:43.401972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146072,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426728,"statfs":{"total":322122547200,"available":318900969472,"internally_reserved":3221225472,"allocated":327680,"data_stored":92382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.716105"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:24:40.701876+0000","last_change":"2020-06-04T08:24:40.701553+0000","last_active":"2020-06-04T08:24:40.701876+0000","last_peered":"2020-06-04T08:24:40.701876+0000","last_clean":"2020-06-04T08:24:40.701876+0000","last_became_active":"2020-06-04T08:24:40.700908+0000","last_became_peered":"2020-06-04T08:24:40.700908+0000","last_unstale":"2020-06-04T08:24:40.701876+0000","last_undegraded":"2020-06-04T08:24:40.701876+0000","last_fullsized":"2020-06-04T08:24:40.701876+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_clean_scrub_stamp":"2020-06-04T08:24:39.674034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:24:40.701698+0000","last_change":"2020-06-04T08:24:40.700964+0000","last_active":"2020-06-04T08:24:40.701698+0000","last_peered":"2020-06-04T08:24:40.701698+0000","last_clean":"2020-06-04T08:24:40.701698+0000","last_became_active":"2020-06-04T08:24:40.700572+0000","last_became_peered":"2020-06-04T08:24:40.700572+0000","last_unstale":"2020-06-04T08:24:40.701698+0000","last_undegraded":"2020-06-04T08:24:40.701698+0000","last_fullsized":"2020-06-04T08:24:40.701698+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_clean_scrub_stamp":"2020-06-04T08:24:39.674034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:24:40.705810+0000","last_change":"2020-06-04T08:24:40.705561+0000","last_active":"2020-06-04T08:24:40.705810+0000","last_peered":"2020-06-04T08:24:40.705810+0000","last_clean":"2020-06-04T08:24:40.705810+0000","last_became_active":"2020-06-04T08:24:40.705158+0000","last_became_peered":"2020-06-04T08:24:40.705158+0000","last_unstale":"2020-06-04T08:24:40.705810+0000","last_undegraded":"2020-06-04T08:24:40.705810+0000","last_fullsized":"2020-06-04T08:24:40.705810+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_clean_scrub_stamp":"2020-06-04T08:24:39.674034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:24:40.697549+0000","last_change":"2020-06-04T08:24:40.697549+0000","last_active":"2020-06-04T08:24:40.697549+0000","last_peered":"2020-06-04T08:24:40.697549+0000","last_clean":"2020-06-04T08:24:40.697549+0000","last_became_active":"2020-06-04T08:24:40.696368+0000","last_became_peered":"2020-06-04T08:24:40.696368+0000","last_unstale":"2020-06-04T08:24:40.697549+0000","last_undegraded":"2020-06-04T08:24:40.697549+0000","last_fullsized":"2020-06-04T08:24:40.697549+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_clean_scrub_stamp":"2020-06-04T08:24:39.674034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:24:40.697043+0000","last_change":"2020-06-04T08:24:36.503370+0000","last_active":"2020-06-04T08:24:40.697043+0000","last_peered":"2020-06-04T08:24:40.697043+0000","last_clean":"2020-06-04T08:24:40.697043+0000","last_became_active":"2020-06-04T08:24:36.501559+0000","last_became_peered":"2020-06-04T08:24:36.501559+0000","last_unstale":"2020-06-04T08:24:40.697043+0000","last_undegraded":"2020-06-04T08:24:40.697043+0000","last_fullsized":"2020-06-04T08:24:40.697043+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:24:33.541130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:24:33.541130+0000","last_clean_scrub_stamp":"2020-06-04T08:24:33.541130+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:24:40.699952+0000","last_change":"2020-06-04T08:24:40.699507+0000","last_active":"2020-06-04T08:24:40.699952+0000","last_peered":"2020-06-04T08:24:40.699952+0000","last_clean":"2020-06-04T08:24:40.699952+0000","last_became_active":"2020-06-04T08:24:40.698422+0000","last_became_peered":"2020-06-04T08:24:40.698422+0000","last_unstale":"2020-06-04T08:24:40.699952+0000","last_undegraded":"2020-06-04T08:24:40.699952+0000","last_fullsized":"2020-06-04T08:24:40.699952+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_clean_scrub_stamp":"2020-06-04T08:24:39.674034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:24:40.698061+0000","last_change":"2020-06-04T08:24:40.698061+0000","last_active":"2020-06-04T08:24:40.698061+0000","last_peered":"2020-06-04T08:24:40.698061+0000","last_clean":"2020-06-04T08:24:40.698061+0000","last_became_active":"2020-06-04T08:24:40.697422+0000","last_became_peered":"2020-06-04T08:24:40.697422+0000","last_unstale":"2020-06-04T08:24:40.698061+0000","last_undegraded":"2020-06-04T08:24:40.698061+0000","last_fullsized":"2020-06-04T08:24:40.698061+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_clean_scrub_stamp":"2020-06-04T08:24:39.674034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:24:40.703389+0000","last_change":"2020-06-04T08:24:40.703389+0000","last_active":"2020-06-04T08:24:40.703389+0000","last_peered":"2020-06-04T08:24:40.703389+0000","last_clean":"2020-06-04T08:24:40.703389+0000","last_became_active":"2020-06-04T08:24:40.702775+0000","last_became_peered":"2020-06-04T08:24:40.702775+0000","last_unstale":"2020-06-04T08:24:40.703389+0000","last_undegraded":"2020-06-04T08:24:40.703389+0000","last_fullsized":"2020-06-04T08:24:40.703389+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_clean_scrub_stamp":"2020-06-04T08:24:39.674034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:24:40.701889+0000","last_change":"2020-06-04T08:24:40.701343+0000","last_active":"2020-06-04T08:24:40.701889+0000","last_peered":"2020-06-04T08:24:40.701889+0000","last_clean":"2020-06-04T08:24:40.701889+0000","last_became_active":"2020-06-04T08:24:40.700339+0000","last_became_peered":"2020-06-04T08:24:40.700339+0000","last_unstale":"2020-06-04T08:24:40.701889+0000","last_undegraded":"2020-06-04T08:24:40.701889+0000","last_fullsized":"2020-06-04T08:24:40.701889+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_clean_scrub_stamp":"2020-06-04T08:24:39.674034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.294}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3740000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6439999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.157}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6400000000000001}]}]}]}} 2020-06-04T08:24:45.371 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2020-06-04T08:24:45.643 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:45.646 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2020-06-04T08:24:45.656 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-06-04T08:24:45.402564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146072,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426728,"statfs":{"total":322122547200,"available":318900969472,"internally_reserved":3221225472,"allocated":327680,"data_stored":92382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.716697"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:24:40.701876+0000","last_change":"2020-06-04T08:24:40.701553+0000","last_active":"2020-06-04T08:24:40.701876+0000","last_peered":"2020-06-04T08:24:40.701876+0000","last_clean":"2020-06-04T08:24:40.701876+0000","last_became_active":"2020-06-04T08:24:40.700908+0000","last_became_peered":"2020-06-04T08:24:40.700908+0000","last_unstale":"2020-06-04T08:24:40.701876+0000","last_undegraded":"2020-06-04T08:24:40.701876+0000","last_fullsized":"2020-06-04T08:24:40.701876+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_clean_scrub_stamp":"2020-06-04T08:24:39.674034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:24:40.701698+0000","last_change":"2020-06-04T08:24:40.700964+0000","last_active":"2020-06-04T08:24:40.701698+0000","last_peered":"2020-06-04T08:24:40.701698+0000","last_clean":"2020-06-04T08:24:40.701698+0000","last_became_active":"2020-06-04T08:24:40.700572+0000","last_became_peered":"2020-06-04T08:24:40.700572+0000","last_unstale":"2020-06-04T08:24:40.701698+0000","last_undegraded":"2020-06-04T08:24:40.701698+0000","last_fullsized":"2020-06-04T08:24:40.701698+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_clean_scrub_stamp":"2020-06-04T08:24:39.674034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:24:40.705810+0000","last_change":"2020-06-04T08:24:40.705561+0000","last_active":"2020-06-04T08:24:40.705810+0000","last_peered":"2020-06-04T08:24:40.705810+0000","last_clean":"2020-06-04T08:24:40.705810+0000","last_became_active":"2020-06-04T08:24:40.705158+0000","last_became_peered":"2020-06-04T08:24:40.705158+0000","last_unstale":"2020-06-04T08:24:40.705810+0000","last_undegraded":"2020-06-04T08:24:40.705810+0000","last_fullsized":"2020-06-04T08:24:40.705810+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_clean_scrub_stamp":"2020-06-04T08:24:39.674034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:24:40.697549+0000","last_change":"2020-06-04T08:24:40.697549+0000","last_active":"2020-06-04T08:24:40.697549+0000","last_peered":"2020-06-04T08:24:40.697549+0000","last_clean":"2020-06-04T08:24:40.697549+0000","last_became_active":"2020-06-04T08:24:40.696368+0000","last_became_peered":"2020-06-04T08:24:40.696368+0000","last_unstale":"2020-06-04T08:24:40.697549+0000","last_undegraded":"2020-06-04T08:24:40.697549+0000","last_fullsized":"2020-06-04T08:24:40.697549+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_clean_scrub_stamp":"2020-06-04T08:24:39.674034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:24:40.697043+0000","last_change":"2020-06-04T08:24:36.503370+0000","last_active":"2020-06-04T08:24:40.697043+0000","last_peered":"2020-06-04T08:24:40.697043+0000","last_clean":"2020-06-04T08:24:40.697043+0000","last_became_active":"2020-06-04T08:24:36.501559+0000","last_became_peered":"2020-06-04T08:24:36.501559+0000","last_unstale":"2020-06-04T08:24:40.697043+0000","last_undegraded":"2020-06-04T08:24:40.697043+0000","last_fullsized":"2020-06-04T08:24:40.697043+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:24:33.541130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:24:33.541130+0000","last_clean_scrub_stamp":"2020-06-04T08:24:33.541130+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:24:40.699952+0000","last_change":"2020-06-04T08:24:40.699507+0000","last_active":"2020-06-04T08:24:40.699952+0000","last_peered":"2020-06-04T08:24:40.699952+0000","last_clean":"2020-06-04T08:24:40.699952+0000","last_became_active":"2020-06-04T08:24:40.698422+0000","last_became_peered":"2020-06-04T08:24:40.698422+0000","last_unstale":"2020-06-04T08:24:40.699952+0000","last_undegraded":"2020-06-04T08:24:40.699952+0000","last_fullsized":"2020-06-04T08:24:40.699952+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_clean_scrub_stamp":"2020-06-04T08:24:39.674034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:24:40.698061+0000","last_change":"2020-06-04T08:24:40.698061+0000","last_active":"2020-06-04T08:24:40.698061+0000","last_peered":"2020-06-04T08:24:40.698061+0000","last_clean":"2020-06-04T08:24:40.698061+0000","last_became_active":"2020-06-04T08:24:40.697422+0000","last_became_peered":"2020-06-04T08:24:40.697422+0000","last_unstale":"2020-06-04T08:24:40.698061+0000","last_undegraded":"2020-06-04T08:24:40.698061+0000","last_fullsized":"2020-06-04T08:24:40.698061+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_clean_scrub_stamp":"2020-06-04T08:24:39.674034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:24:40.703389+0000","last_change":"2020-06-04T08:24:40.703389+0000","last_active":"2020-06-04T08:24:40.703389+0000","last_peered":"2020-06-04T08:24:40.703389+0000","last_clean":"2020-06-04T08:24:40.703389+0000","last_became_active":"2020-06-04T08:24:40.702775+0000","last_became_peered":"2020-06-04T08:24:40.702775+0000","last_unstale":"2020-06-04T08:24:40.703389+0000","last_undegraded":"2020-06-04T08:24:40.703389+0000","last_fullsized":"2020-06-04T08:24:40.703389+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_clean_scrub_stamp":"2020-06-04T08:24:39.674034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:24:40.701889+0000","last_change":"2020-06-04T08:24:40.701343+0000","last_active":"2020-06-04T08:24:40.701889+0000","last_peered":"2020-06-04T08:24:40.701889+0000","last_clean":"2020-06-04T08:24:40.701889+0000","last_became_active":"2020-06-04T08:24:40.700339+0000","last_became_peered":"2020-06-04T08:24:40.700339+0000","last_unstale":"2020-06-04T08:24:40.701889+0000","last_undegraded":"2020-06-04T08:24:40.701889+0000","last_fullsized":"2020-06-04T08:24:40.701889+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:24:39.674034+0000","last_clean_scrub_stamp":"2020-06-04T08:24:39.674034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.294}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3740000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6439999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.157}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6400000000000001}]}]}]}} 2020-06-04T08:24:45.657 INFO:tasks.ceph.ceph_manager.c1:clean! 2020-06-04T08:24:45.657 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2020-06-04T08:24:45.658 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2020-06-04T08:24:45.658 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early health --format=json 2020-06-04T08:24:45.967 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:45.984 INFO:teuthology.orchestra.run.smithi060.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-06-04T08:24:45.985 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2020-06-04T08:24:45.985 INFO:teuthology.run_tasks:Running task ceph... 2020-06-04T08:24:46.021 INFO:tasks.ceph:Creating ceph cluster c2... 2020-06-04T08:24:46.022 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2020-06-04T08:24:46.023 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuvalif-2020-06-04_08:07:49-rgw:multisite-notifications_persistent_delivery2-distro-basic-smithi/5116651', 'branch': 'notifications_persistent_delivery2', 'description': 'rgw:multisite/{clusters.yaml frontend/beast.yaml omap_limits.yaml overrides.yaml realms/three-zone-plus-pubsub.yaml tasks/test_multi.yaml valgrind.yaml}', 'email': None, 'first_in_suite': False, 'job_id': '5116651', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuvalif-2020-06-04_08:07:49-rgw:multisite-notifications_persistent_delivery2-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'notifications_persistent_delivery2'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': '5a141c6290e05d95ab4ea2df3c0cb561d6b8e984', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '5a141c6290e05d95ab4ea2df3c0cb561d6b8e984'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'notifications_persistent_delivery2', 'sha1': '5a141c6290e05d95ab4ea2df3c0cb561d6b8e984'}}, 'owner': 'scheduled_yuvalif@teuthology', 'priority': 10, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '5a141c6290e05d95ab4ea2df3c0cb561d6b8e984', 'sleep_before_teardown': 0, 'suite': 'rgw:multisite', 'suite_branch': 'notifications_persistent_delivery2', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_notifications_persistent_delivery2/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '5a141c6290e05d95ab4ea2df3c0cb561d6b8e984', 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': '5a141c6290e05d95ab4ea2df3c0cb561d6b8e984', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'sha1': '5a141c6290e05d95ab4ea2df3c0cb561d6b8e984', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'tube': 'smithi', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.27832', 'targets': {'smithi060.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2sBIojSjMzey9aOmMk+qXyf+3XXUKYC8z2s2P34SimG2IxQrlBKbYii8YwiPqmDxwHXFwGe6emOAHi2Qbt5swScPnApnwLaqCisyP+CRaR48Oe1PgL/nV054P9J1dYeMbKnVdGwSABk6R3cM6e5Na3et3aDMH3L/ftTdxo4wToKWEbQCCqpgK4s2TEbKtGQ6CBuNtHtqBoV65ni853lz6jOLNfIc1RDcaNCeMeJsUrtggxjrQ8j/wxjjHqzw4PbszWwbv8sF6WBsFKQFcoA4CYliBVWzL5/BDK+FTMk0kAG4YX3FV4YYH+yPBFuAxLBqdkx5nfegBuP4xHSHpJ31b', 'smithi196.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDejCVS35mBeiMVuY30POoD2qFLttOeNDZSkW0Byb9L8vUr1j0Thf72EQ2VqZ0FKgkgkADDw2bY7nNUZYxRfjTGLJNwr66E8oNgT82C44Muwli1+YU7JIkaFAVNPxrE28N1dOjdApLLT9i0W8pQOifHPKlvMWoAPMm6V8U5RDbTWdQ6gV0oq1SPlO+PNgCmV8UG7YQ02w/YXJcRgS0mNGRGqSJi1UMaVwQsZvNv4+OI6hOzjP5kxmpt7k2ThLv4CmmlzHfE6Pzb/Tv7rw9R4Coq/EFsTuzSbt2NShPUbgNX66BDX7kNZmYt8p372zyYfQZGUjDzyXXDC1eSuNiBJVhn'}} 2020-06-04T08:24:46.024 INFO:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-06-04T08:24:46.028 INFO:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2020-06-04T08:24:46.040 INFO:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/run/ceph 2020-06-04T08:24:46.075 INFO:teuthology.orchestra.run.smithi196:> sudo install -d -m0777 -- /var/run/ceph 2020-06-04T08:24:46.147 DEBUG:teuthology.orchestra.remote:smithi196:/scratch_devs is 72B 2020-06-04T08:24:46.161 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2020-06-04T08:24:46.162 INFO:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-06-04T08:24:46.169 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2020-06-04T08:24:46.170 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-04T08:24:46.170 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 6h/6d Inode: 341 Links: 1 2020-06-04T08:24:46.170 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-04T08:24:46.171 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2020-06-04 08:21:57.923768326 +0000 2020-06-04T08:24:46.171 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2020-06-04 08:21:57.819716330 +0000 2020-06-04T08:24:46.171 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2020-06-04 08:21:57.819716330 +0000 2020-06-04T08:24:46.174 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: - 2020-06-04T08:24:46.179 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2020-06-04T08:24:46.180 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2020-06-04T08:24:46.180 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000389124 s, 1.3 MB/s 2020-06-04T08:24:46.187 INFO:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-06-04T08:24:46.235 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2020-06-04T08:24:46.237 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-04T08:24:46.237 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 6h/6d Inode: 348 Links: 1 2020-06-04T08:24:46.237 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-04T08:24:46.238 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2020-06-04 08:21:58.179896319 +0000 2020-06-04T08:24:46.238 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2020-06-04 08:21:58.063838322 +0000 2020-06-04T08:24:46.238 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2020-06-04 08:21:58.063838322 +0000 2020-06-04T08:24:46.239 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: - 2020-06-04T08:24:46.245 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2020-06-04T08:24:46.245 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2020-06-04T08:24:46.245 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000268206 s, 1.9 MB/s 2020-06-04T08:24:46.267 INFO:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-06-04T08:24:46.317 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-06-04T08:24:46.318 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-04T08:24:46.318 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 6h/6d Inode: 354 Links: 1 2020-06-04T08:24:46.318 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-04T08:24:46.319 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2020-06-04 08:21:58.408010314 +0000 2020-06-04T08:24:46.319 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2020-06-04 08:21:58.303958315 +0000 2020-06-04T08:24:46.319 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2020-06-04 08:21:58.303958315 +0000 2020-06-04T08:24:46.319 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: - 2020-06-04T08:24:46.327 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2020-06-04T08:24:46.328 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2020-06-04T08:24:46.328 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.00035515 s, 1.4 MB/s 2020-06-04T08:24:46.332 INFO:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-06-04T08:24:46.381 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2020-06-04T08:24:46.382 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-06-04T08:24:46.382 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 6h/6d Inode: 360 Links: 1 2020-06-04T08:24:46.382 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-06-04T08:24:46.383 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2020-06-04 08:21:58.672142305 +0000 2020-06-04T08:24:46.383 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2020-06-04 08:21:58.536074309 +0000 2020-06-04T08:24:46.383 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2020-06-04 08:21:58.536074309 +0000 2020-06-04T08:24:46.383 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: - 2020-06-04T08:24:46.393 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2020-06-04T08:24:46.393 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2020-06-04T08:24:46.394 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000217389 s, 2.4 MB/s 2020-06-04T08:24:46.397 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-06-04T08:24:46.398 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi196.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-06-04T08:24:46.398 INFO:tasks.ceph:Generating config... 2020-06-04T08:24:46.400 INFO:tasks.ceph:[client] debug rgw = 20 2020-06-04T08:24:46.400 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2020-06-04T08:24:46.401 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2020-06-04T08:24:46.401 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-06-04T08:24:46.402 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2020-06-04T08:24:46.402 INFO:tasks.ceph:[client] rgw data log num shards = 4 2020-06-04T08:24:46.402 INFO:tasks.ceph:[client] rgw md log max shards = 4 2020-06-04T08:24:46.403 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2020-06-04T08:24:46.403 INFO:tasks.ceph:[client] setgroup = ceph 2020-06-04T08:24:46.403 INFO:tasks.ceph:[client] setuser = ceph 2020-06-04T08:24:46.404 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2020-06-04T08:24:46.404 INFO:tasks.ceph:[mgr] debug mgr = 20 2020-06-04T08:24:46.405 INFO:tasks.ceph:[mgr] debug ms = 1 2020-06-04T08:24:46.406 INFO:tasks.ceph:[mon] debug mon = 20 2020-06-04T08:24:46.406 INFO:tasks.ceph:[mon] debug ms = 1 2020-06-04T08:24:46.407 INFO:tasks.ceph:[mon] debug paxos = 20 2020-06-04T08:24:46.407 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2020-06-04T08:24:46.408 INFO:tasks.ceph:[osd] debug filestore = 20 2020-06-04T08:24:46.408 INFO:tasks.ceph:[osd] debug journal = 20 2020-06-04T08:24:46.408 INFO:tasks.ceph:[osd] debug ms = 20 2020-06-04T08:24:46.409 INFO:tasks.ceph:[osd] debug osd = 25 2020-06-04T08:24:46.409 INFO:tasks.ceph:[osd] osd fast shutdown = False 2020-06-04T08:24:46.409 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2020-06-04T08:24:46.410 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2020-06-04T08:24:46.410 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2020-06-04T08:24:46.410 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2020-06-04T08:24:46.411 INFO:tasks.ceph:Setting up c2.mon.a... 2020-06-04T08:24:46.411 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2020-06-04T08:24:46.471 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/c2.keyring 2020-06-04T08:24:46.477 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2020-06-04T08:24:46.553 INFO:teuthology.orchestra.run.smithi196:> sudo chmod 0644 /etc/ceph/c2.keyring 2020-06-04T08:24:46.609 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.196')] 2020-06-04T08:24:46.609 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.196 --print /home/ubuntu/cephtest/c2.monmap 2020-06-04T08:24:46.681 INFO:teuthology.orchestra.run.smithi196.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2020-06-04T08:24:46.682 INFO:teuthology.orchestra.run.smithi196.stdout:monmaptool: generated fsid f2c8e8e3-2947-4b15-b4d7-3d00a5ceca6c 2020-06-04T08:24:46.682 INFO:teuthology.orchestra.run.smithi196.stdout:epoch 0 2020-06-04T08:24:46.683 INFO:teuthology.orchestra.run.smithi196.stdout:fsid f2c8e8e3-2947-4b15-b4d7-3d00a5ceca6c 2020-06-04T08:24:46.683 INFO:teuthology.orchestra.run.smithi196.stdout:last_changed 2020-06-04T08:24:46.681089+0000 2020-06-04T08:24:46.683 INFO:teuthology.orchestra.run.smithi196.stdout:created 2020-06-04T08:24:46.681089+0000 2020-06-04T08:24:46.684 INFO:teuthology.orchestra.run.smithi196.stdout:min_mon_release 0 (unknown) 2020-06-04T08:24:46.684 INFO:teuthology.orchestra.run.smithi196.stdout:0: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.a 2020-06-04T08:24:46.684 INFO:teuthology.orchestra.run.smithi196.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2020-06-04T08:24:46.686 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID f2c8e8e3-2947-4b15-b4d7-3d00a5ceca6c... 2020-06-04T08:24:46.688 INFO:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-06-04T08:24:46.692 INFO:teuthology.orchestra.run.smithi196:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2020-06-04T08:24:46.733 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2020-06-04T08:24:46.734 INFO:teuthology.orchestra.run.smithi060.stdout: chdir = "" 2020-06-04T08:24:46.734 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-06-04T08:24:46.735 INFO:teuthology.orchestra.run.smithi060.stdout: auth supported = cephx 2020-06-04T08:24:46.735 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.735 INFO:teuthology.orchestra.run.smithi060.stdout: filestore xattr use omap = true 2020-06-04T08:24:46.736 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.736 INFO:teuthology.orchestra.run.smithi060.stdout: mon clock drift allowed = 1.000 2020-06-04T08:24:46.736 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.736 INFO:teuthology.orchestra.run.smithi060.stdout: osd crush chooseleaf type = 0 2020-06-04T08:24:46.736 INFO:teuthology.orchestra.run.smithi060.stdout: auth debug = true 2020-06-04T08:24:46.737 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.737 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on old message = true 2020-06-04T08:24:46.737 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on bug = true 2020-06-04T08:24:46.737 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.738 INFO:teuthology.orchestra.run.smithi060.stdout: mon max pg per osd = 10000 # >= luminous 2020-06-04T08:24:46.738 INFO:teuthology.orchestra.run.smithi060.stdout: mon pg warn max object skew = 0 2020-06-04T08:24:46.738 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.738 INFO:teuthology.orchestra.run.smithi060.stdout: # disable pg_autoscaler by default for new pools 2020-06-04T08:24:46.738 INFO:teuthology.orchestra.run.smithi060.stdout: osd_pool_default_pg_autoscale_mode = off 2020-06-04T08:24:46.739 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.739 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default size = 2 2020-06-04T08:24:46.739 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.739 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow primary affinity = true 2020-06-04T08:24:46.739 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow pg remap = true 2020-06-04T08:24:46.740 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on legacy crush tunables = false 2020-06-04T08:24:46.740 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on crush straw calc version zero = false 2020-06-04T08:24:46.741 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on no sortbitwise = false 2020-06-04T08:24:46.741 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on osd down out interval zero = false 2020-06-04T08:24:46.741 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on too few osds = false 2020-06-04T08:24:46.741 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-06-04T08:24:46.742 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_no_redundancy = false 2020-06-04T08:24:46.742 INFO:teuthology.orchestra.run.smithi060.stdout: mon_allow_pool_size_one = true 2020-06-04T08:24:46.742 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.742 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-06-04T08:24:46.743 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.743 INFO:teuthology.orchestra.run.smithi060.stdout: osd default data pool replay window = 5 2020-06-04T08:24:46.743 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.743 INFO:teuthology.orchestra.run.smithi060.stdout: mon allow pool delete = true 2020-06-04T08:24:46.744 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.744 INFO:teuthology.orchestra.run.smithi060.stdout: mon cluster log file level = debug 2020-06-04T08:24:46.744 INFO:teuthology.orchestra.run.smithi060.stdout: debug asserts on shutdown = true 2020-06-04T08:24:46.744 INFO:teuthology.orchestra.run.smithi060.stdout: mon host = 172.21.15.196 2020-06-04T08:24:46.744 INFO:teuthology.orchestra.run.smithi060.stdout: osd heartbeat grace = 40 2020-06-04T08:24:46.744 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = f2c8e8e3-2947-4b15-b4d7-3d00a5ceca6c 2020-06-04T08:24:46.745 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.745 INFO:teuthology.orchestra.run.smithi060.stdout:[osd] 2020-06-04T08:24:46.745 INFO:teuthology.orchestra.run.smithi060.stdout: osd journal size = 100 2020-06-04T08:24:46.748 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.748 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub load threshold = 5.0 2020-06-04T08:24:46.748 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub max interval = 600 2020-06-04T08:24:46.748 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.748 INFO:teuthology.orchestra.run.smithi060.stdout: osd recover clone overlap = true 2020-06-04T08:24:46.749 INFO:teuthology.orchestra.run.smithi060.stdout: osd recovery max chunk = 1048576 2020-06-04T08:24:46.749 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.749 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug shutdown = true 2020-06-04T08:24:46.749 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug op order = true 2020-06-04T08:24:46.750 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify stray on activate = true 2020-06-04T08:24:46.750 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.751 INFO:teuthology.orchestra.run.smithi060.stdout: osd open classes on start = true 2020-06-04T08:24:46.751 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug pg log writeout = true 2020-06-04T08:24:46.752 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.752 INFO:teuthology.orchestra.run.smithi060.stdout: osd deep scrub update digest min age = 30 2020-06-04T08:24:46.752 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.753 INFO:teuthology.orchestra.run.smithi060.stdout: osd map max advance = 10 2020-06-04T08:24:46.753 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.753 INFO:teuthology.orchestra.run.smithi060.stdout: journal zero on create = true 2020-06-04T08:24:46.754 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.754 INFO:teuthology.orchestra.run.smithi060.stdout: filestore ondisk finisher threads = 3 2020-06-04T08:24:46.754 INFO:teuthology.orchestra.run.smithi060.stdout: filestore apply finisher threads = 3 2020-06-04T08:24:46.755 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.755 INFO:teuthology.orchestra.run.smithi060.stdout: bdev debug aio = true 2020-06-04T08:24:46.755 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug misdirected ops = true 2020-06-04T08:24:46.756 INFO:teuthology.orchestra.run.smithi060.stdout: debug filestore = 20 2020-06-04T08:24:46.756 INFO:teuthology.orchestra.run.smithi060.stdout: debug journal = 20 2020-06-04T08:24:46.756 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 20 2020-06-04T08:24:46.757 INFO:teuthology.orchestra.run.smithi060.stdout: debug osd = 25 2020-06-04T08:24:46.757 INFO:teuthology.orchestra.run.smithi060.stdout: osd fast shutdown = False 2020-06-04T08:24:46.757 INFO:teuthology.orchestra.run.smithi060.stdout: rocksdb delete range threshold = 0 2020-06-04T08:24:46.758 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.758 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr] 2020-06-04T08:24:46.758 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2020-06-04T08:24:46.759 INFO:teuthology.orchestra.run.smithi060.stdout: debug mgr = 20 2020-06-04T08:24:46.759 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2020-06-04T08:24:46.759 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2020-06-04T08:24:46.760 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2020-06-04T08:24:46.760 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2020-06-04T08:24:46.760 INFO:teuthology.orchestra.run.smithi060.stdout: mgr/telemetry/nag = false 2020-06-04T08:24:46.761 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.762 INFO:teuthology.orchestra.run.smithi060.stdout:[mon] 2020-06-04T08:24:46.762 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2020-06-04T08:24:46.762 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2020-06-04T08:24:46.762 INFO:teuthology.orchestra.run.smithi060.stdout: debug paxos = 20 2020-06-04T08:24:46.763 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2020-06-04T08:24:46.763 INFO:teuthology.orchestra.run.smithi060.stdout: mon data avail warn = 5 2020-06-04T08:24:46.763 INFO:teuthology.orchestra.run.smithi060.stdout: mon mgr mkfs grace = 240 2020-06-04T08:24:46.763 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2020-06-04T08:24:46.763 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd reporter subtree level = osd 2020-06-04T08:24:46.764 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd prime pg temp = true 2020-06-04T08:24:46.764 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2020-06-04T08:24:46.764 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd crush smoke test = False 2020-06-04T08:24:46.764 INFO:teuthology.orchestra.run.smithi060.stdout: 2020-06-04T08:24:46.764 INFO:teuthology.orchestra.run.smithi060.stdout:[client] 2020-06-04T08:24:46.764 INFO:teuthology.orchestra.run.smithi060.stdout: rgw cache enabled = true 2020-06-04T08:24:46.765 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable ops log = true 2020-06-04T08:24:46.765 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable usage log = true 2020-06-04T08:24:46.765 INFO:teuthology.orchestra.run.smithi060.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-06-04T08:24:46.765 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-06-04T08:24:46.766 INFO:teuthology.orchestra.run.smithi060.stdout: debug rgw = 20 2020-06-04T08:24:46.766 INFO:teuthology.orchestra.run.smithi060.stdout: rgw crypt require ssl = False 2020-06-04T08:24:46.766 INFO:teuthology.orchestra.run.smithi060.stdout: rgw crypt s3 kms backend = testing 2020-06-04T08:24:46.766 INFO:teuthology.orchestra.run.smithi060.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-06-04T08:24:46.767 INFO:teuthology.orchestra.run.smithi060.stdout: rgw curl low speed time = 300 2020-06-04T08:24:46.767 INFO:teuthology.orchestra.run.smithi060.stdout: rgw data log num shards = 4 2020-06-04T08:24:46.767 INFO:teuthology.orchestra.run.smithi060.stdout: rgw md log max shards = 4 2020-06-04T08:24:46.767 INFO:teuthology.orchestra.run.smithi060.stdout: rgw sync log trim interval = 0 2020-06-04T08:24:46.767 INFO:teuthology.orchestra.run.smithi060.stdout: setgroup = ceph 2020-06-04T08:24:46.767 INFO:teuthology.orchestra.run.smithi060.stdout: setuser = ceph 2020-06-04T08:24:46.768 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.a] 2020-06-04T08:24:46.768 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.0] 2020-06-04T08:24:46.768 INFO:teuthology.orchestra.run.smithi060.stdout: osd_max_omap_entries_per_request = 10 2020-06-04T08:24:46.768 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.1] 2020-06-04T08:24:46.768 INFO:teuthology.orchestra.run.smithi060.stdout: osd_max_omap_entries_per_request = 1000 2020-06-04T08:24:46.769 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.2] 2020-06-04T08:24:46.769 INFO:teuthology.orchestra.run.smithi060.stdout: osd_max_omap_entries_per_request = 10000 2020-06-04T08:24:46.770 INFO:teuthology.orchestra.run.smithi196.stdout:[global] 2020-06-04T08:24:46.771 INFO:teuthology.orchestra.run.smithi196.stdout: chdir = "" 2020-06-04T08:24:46.771 INFO:teuthology.orchestra.run.smithi196.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-06-04T08:24:46.771 INFO:teuthology.orchestra.run.smithi196.stdout: auth supported = cephx 2020-06-04T08:24:46.771 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.771 INFO:teuthology.orchestra.run.smithi196.stdout: filestore xattr use omap = true 2020-06-04T08:24:46.771 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.772 INFO:teuthology.orchestra.run.smithi196.stdout: mon clock drift allowed = 1.000 2020-06-04T08:24:46.772 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.772 INFO:teuthology.orchestra.run.smithi196.stdout: osd crush chooseleaf type = 0 2020-06-04T08:24:46.772 INFO:teuthology.orchestra.run.smithi196.stdout: auth debug = true 2020-06-04T08:24:46.772 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.773 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on old message = true 2020-06-04T08:24:46.773 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on bug = true 2020-06-04T08:24:46.773 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.773 INFO:teuthology.orchestra.run.smithi196.stdout: mon max pg per osd = 10000 # >= luminous 2020-06-04T08:24:46.773 INFO:teuthology.orchestra.run.smithi196.stdout: mon pg warn max object skew = 0 2020-06-04T08:24:46.774 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.774 INFO:teuthology.orchestra.run.smithi196.stdout: # disable pg_autoscaler by default for new pools 2020-06-04T08:24:46.774 INFO:teuthology.orchestra.run.smithi196.stdout: osd_pool_default_pg_autoscale_mode = off 2020-06-04T08:24:46.774 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.774 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default size = 2 2020-06-04T08:24:46.774 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.775 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow primary affinity = true 2020-06-04T08:24:46.775 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow pg remap = true 2020-06-04T08:24:46.775 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on legacy crush tunables = false 2020-06-04T08:24:46.775 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on crush straw calc version zero = false 2020-06-04T08:24:46.775 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on no sortbitwise = false 2020-06-04T08:24:46.776 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on osd down out interval zero = false 2020-06-04T08:24:46.776 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on too few osds = false 2020-06-04T08:24:46.776 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-06-04T08:24:46.776 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_no_redundancy = false 2020-06-04T08:24:46.776 INFO:teuthology.orchestra.run.smithi196.stdout: mon_allow_pool_size_one = true 2020-06-04T08:24:46.777 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.777 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-06-04T08:24:46.777 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.777 INFO:teuthology.orchestra.run.smithi196.stdout: osd default data pool replay window = 5 2020-06-04T08:24:46.777 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.777 INFO:teuthology.orchestra.run.smithi196.stdout: mon allow pool delete = true 2020-06-04T08:24:46.778 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.778 INFO:teuthology.orchestra.run.smithi196.stdout: mon cluster log file level = debug 2020-06-04T08:24:46.778 INFO:teuthology.orchestra.run.smithi196.stdout: debug asserts on shutdown = true 2020-06-04T08:24:46.779 INFO:teuthology.orchestra.run.smithi196.stdout: mon host = 172.21.15.196 2020-06-04T08:24:46.779 INFO:teuthology.orchestra.run.smithi196.stdout: osd heartbeat grace = 40 2020-06-04T08:24:46.779 INFO:teuthology.orchestra.run.smithi196.stdout: fsid = f2c8e8e3-2947-4b15-b4d7-3d00a5ceca6c 2020-06-04T08:24:46.779 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.779 INFO:teuthology.orchestra.run.smithi196.stdout:[osd] 2020-06-04T08:24:46.780 INFO:teuthology.orchestra.run.smithi196.stdout: osd journal size = 100 2020-06-04T08:24:46.780 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.780 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub load threshold = 5.0 2020-06-04T08:24:46.780 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub max interval = 600 2020-06-04T08:24:46.780 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.781 INFO:teuthology.orchestra.run.smithi196.stdout: osd recover clone overlap = true 2020-06-04T08:24:46.781 INFO:teuthology.orchestra.run.smithi196.stdout: osd recovery max chunk = 1048576 2020-06-04T08:24:46.781 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.781 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug shutdown = true 2020-06-04T08:24:46.781 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug op order = true 2020-06-04T08:24:46.782 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug verify stray on activate = true 2020-06-04T08:24:46.782 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.782 INFO:teuthology.orchestra.run.smithi196.stdout: osd open classes on start = true 2020-06-04T08:24:46.782 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug pg log writeout = true 2020-06-04T08:24:46.782 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.782 INFO:teuthology.orchestra.run.smithi196.stdout: osd deep scrub update digest min age = 30 2020-06-04T08:24:46.783 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.783 INFO:teuthology.orchestra.run.smithi196.stdout: osd map max advance = 10 2020-06-04T08:24:46.783 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.783 INFO:teuthology.orchestra.run.smithi196.stdout: journal zero on create = true 2020-06-04T08:24:46.783 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.784 INFO:teuthology.orchestra.run.smithi196.stdout: filestore ondisk finisher threads = 3 2020-06-04T08:24:46.784 INFO:teuthology.orchestra.run.smithi196.stdout: filestore apply finisher threads = 3 2020-06-04T08:24:46.784 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.784 INFO:teuthology.orchestra.run.smithi196.stdout: bdev debug aio = true 2020-06-04T08:24:46.784 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug misdirected ops = true 2020-06-04T08:24:46.784 INFO:teuthology.orchestra.run.smithi196.stdout: debug filestore = 20 2020-06-04T08:24:46.785 INFO:teuthology.orchestra.run.smithi196.stdout: debug journal = 20 2020-06-04T08:24:46.785 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 20 2020-06-04T08:24:46.785 INFO:teuthology.orchestra.run.smithi196.stdout: debug osd = 25 2020-06-04T08:24:46.785 INFO:teuthology.orchestra.run.smithi196.stdout: osd fast shutdown = False 2020-06-04T08:24:46.785 INFO:teuthology.orchestra.run.smithi196.stdout: rocksdb delete range threshold = 0 2020-06-04T08:24:46.786 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.786 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr] 2020-06-04T08:24:46.786 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2020-06-04T08:24:46.786 INFO:teuthology.orchestra.run.smithi196.stdout: debug mgr = 20 2020-06-04T08:24:46.786 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2020-06-04T08:24:46.786 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2020-06-04T08:24:46.787 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2020-06-04T08:24:46.787 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2020-06-04T08:24:46.787 INFO:teuthology.orchestra.run.smithi196.stdout: mgr/telemetry/nag = false 2020-06-04T08:24:46.787 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.787 INFO:teuthology.orchestra.run.smithi196.stdout:[mon] 2020-06-04T08:24:46.788 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2020-06-04T08:24:46.788 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2020-06-04T08:24:46.788 INFO:teuthology.orchestra.run.smithi196.stdout: debug paxos = 20 2020-06-04T08:24:46.788 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2020-06-04T08:24:46.788 INFO:teuthology.orchestra.run.smithi196.stdout: mon data avail warn = 5 2020-06-04T08:24:46.788 INFO:teuthology.orchestra.run.smithi196.stdout: mon mgr mkfs grace = 240 2020-06-04T08:24:46.789 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2020-06-04T08:24:46.789 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd reporter subtree level = osd 2020-06-04T08:24:46.789 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd prime pg temp = true 2020-06-04T08:24:46.789 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2020-06-04T08:24:46.789 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd crush smoke test = False 2020-06-04T08:24:46.790 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:24:46.790 INFO:teuthology.orchestra.run.smithi196.stdout:[client] 2020-06-04T08:24:46.790 INFO:teuthology.orchestra.run.smithi196.stdout: rgw cache enabled = true 2020-06-04T08:24:46.790 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable ops log = true 2020-06-04T08:24:46.790 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable usage log = true 2020-06-04T08:24:46.790 INFO:teuthology.orchestra.run.smithi196.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-06-04T08:24:46.791 INFO:teuthology.orchestra.run.smithi196.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-06-04T08:24:46.791 INFO:teuthology.orchestra.run.smithi196.stdout: debug rgw = 20 2020-06-04T08:24:46.791 INFO:teuthology.orchestra.run.smithi196.stdout: rgw crypt require ssl = False 2020-06-04T08:24:46.791 INFO:teuthology.orchestra.run.smithi196.stdout: rgw crypt s3 kms backend = testing 2020-06-04T08:24:46.791 INFO:teuthology.orchestra.run.smithi196.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2020-06-04T08:24:46.792 INFO:teuthology.orchestra.run.smithi196.stdout: rgw curl low speed time = 300 2020-06-04T08:24:46.792 INFO:teuthology.orchestra.run.smithi196.stdout: rgw data log num shards = 4 2020-06-04T08:24:46.792 INFO:teuthology.orchestra.run.smithi196.stdout: rgw md log max shards = 4 2020-06-04T08:24:46.792 INFO:teuthology.orchestra.run.smithi196.stdout: rgw sync log trim interval = 0 2020-06-04T08:24:46.792 INFO:teuthology.orchestra.run.smithi196.stdout: setgroup = ceph 2020-06-04T08:24:46.792 INFO:teuthology.orchestra.run.smithi196.stdout: setuser = ceph 2020-06-04T08:24:46.793 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.a] 2020-06-04T08:24:46.793 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.0] 2020-06-04T08:24:46.793 INFO:teuthology.orchestra.run.smithi196.stdout: osd_max_omap_entries_per_request = 10 2020-06-04T08:24:46.793 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.1] 2020-06-04T08:24:46.793 INFO:teuthology.orchestra.run.smithi196.stdout: osd_max_omap_entries_per_request = 1000 2020-06-04T08:24:46.794 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.2] 2020-06-04T08:24:46.794 INFO:teuthology.orchestra.run.smithi196.stdout: osd_max_omap_entries_per_request = 10000 2020-06-04T08:24:46.794 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2020-06-04T08:24:46.795 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2020-06-04T08:24:46.856 INFO:tasks.ceph:Copying monmap to all nodes... 2020-06-04T08:24:46.865 DEBUG:teuthology.orchestra.remote:smithi196:/etc/ceph/c2.keyring is 206B 2020-06-04T08:24:46.883 DEBUG:teuthology.orchestra.remote:smithi196:/home/ubuntu/cephtest/c2.monmap is 179B 2020-06-04T08:24:46.893 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi060.front.sepia.ceph.com 2020-06-04T08:24:46.893 INFO:teuthology.orchestra.run.smithi060:> sudo sh -c 'cat > /etc/ceph/c2.keyring' && sudo chmod 0644 /etc/ceph/c2.keyring 2020-06-04T08:24:46.949 INFO:teuthology.orchestra.run.smithi060:> cat > /home/ubuntu/cephtest/c2.monmap 2020-06-04T08:24:47.039 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi196.front.sepia.ceph.com 2020-06-04T08:24:47.039 INFO:teuthology.orchestra.run.smithi196:> sudo sh -c 'cat > /etc/ceph/c2.keyring' && sudo chmod 0644 /etc/ceph/c2.keyring 2020-06-04T08:24:47.095 INFO:teuthology.orchestra.run.smithi196:> cat > /home/ubuntu/cephtest/c2.monmap 2020-06-04T08:24:47.184 INFO:tasks.ceph:Setting up mon nodes... 2020-06-04T08:24:47.185 INFO:tasks.ceph:Setting up mgr nodes... 2020-06-04T08:24:47.185 INFO:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2020-06-04T08:24:47.269 INFO:teuthology.orchestra.run.smithi196.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2020-06-04T08:24:47.275 INFO:tasks.ceph:Setting up mds nodes... 2020-06-04T08:24:47.275 INFO:tasks.ceph_client:Setting up client nodes... 2020-06-04T08:24:47.276 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2020-06-04T08:24:47.352 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/c2.client.0.keyring 2020-06-04T08:24:47.363 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2020-06-04T08:24:47.439 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/c2.client.1.keyring 2020-06-04T08:24:47.452 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-06-04T08:24:47.452 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi196.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2020-06-04T08:24:47.453 INFO:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2020-06-04T08:24:47.509 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-06-04T08:24:47.510 INFO:tasks.ceph:role: c2.osd.0 2020-06-04T08:24:47.510 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi196.front.sepia.ceph.com 2020-06-04T08:24:47.511 INFO:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-06-04T08:24:48.035 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-06-04T08:24:48.036 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2020-06-04T08:24:48.036 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-06-04T08:24:48.036 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-06-04T08:24:48.037 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2020-06-04T08:24:48.037 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-06-04T08:24:48.037 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-06-04T08:24:48.037 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-06-04T08:24:48.037 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-06-04T08:24:48.038 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2020-06-04T08:24:48.039 INFO:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2020-06-04T08:24:48.123 INFO:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2020-06-04T08:24:48.135 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: /sbin/restorecon: command not found 2020-06-04T08:24:48.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-06-04T08:24:48.137 INFO:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2020-06-04T08:24:48.197 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-06-04T08:24:48.197 INFO:tasks.ceph:role: c2.osd.1 2020-06-04T08:24:48.197 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi196.front.sepia.ceph.com 2020-06-04T08:24:48.198 INFO:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-06-04T08:24:48.716 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-06-04T08:24:48.717 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2020-06-04T08:24:48.717 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-06-04T08:24:48.717 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-06-04T08:24:48.717 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2020-06-04T08:24:48.718 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-06-04T08:24:48.718 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-06-04T08:24:48.718 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-06-04T08:24:48.718 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-06-04T08:24:48.719 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2020-06-04T08:24:48.720 INFO:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2020-06-04T08:24:48.760 INFO:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2020-06-04T08:24:48.818 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: /sbin/restorecon: command not found 2020-06-04T08:24:48.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-06-04T08:24:48.819 INFO:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2020-06-04T08:24:48.876 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2020-06-04T08:24:48.877 INFO:tasks.ceph:role: c2.osd.2 2020-06-04T08:24:48.877 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi196.front.sepia.ceph.com 2020-06-04T08:24:48.878 INFO:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-06-04T08:24:49.388 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-06-04T08:24:49.389 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2020-06-04T08:24:49.389 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2020-06-04T08:24:49.389 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-06-04T08:24:49.390 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2020-06-04T08:24:49.390 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2020-06-04T08:24:49.390 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-06-04T08:24:49.390 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-06-04T08:24:49.390 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-06-04T08:24:49.391 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2020-06-04T08:24:49.392 INFO:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2020-06-04T08:24:49.435 INFO:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2020-06-04T08:24:49.494 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: /sbin/restorecon: command not found 2020-06-04T08:24:49.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-06-04T08:24:49.496 INFO:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2020-06-04T08:24:49.600 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:24:49.599+0000 7fc928b46d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2020-06-04T08:24:49.601 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:24:49.599+0000 7fc928b46d80 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2020-06-04T08:24:49.602 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:24:49.599+0000 7fc928b46d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-06-04T08:24:49.602 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:24:49.599+0000 7fc928b46d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-06-04T08:24:49.602 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:24:49.599+0000 7fc928b46d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2020-06-04T08:24:49.604 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:24:49.603+0000 7fc928b46d80 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2020-06-04T08:24:51.520 INFO:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2020-06-04T08:24:51.544 INFO:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2020-06-04T08:24:51.627 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:24:51.627+0000 7f35e3e0fd80 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2020-06-04T08:24:51.628 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:24:51.627+0000 7f35e3e0fd80 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2020-06-04T08:24:51.628 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:24:51.627+0000 7f35e3e0fd80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-06-04T08:24:51.628 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:24:51.627+0000 7f35e3e0fd80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-06-04T08:24:51.628 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:24:51.627+0000 7f35e3e0fd80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2020-06-04T08:24:51.629 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:24:51.627+0000 7f35e3e0fd80 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2020-06-04T08:24:53.555 INFO:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2020-06-04T08:24:53.632 INFO:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2020-06-04T08:24:53.696 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:24:53.695+0000 7faf91824d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2020-06-04T08:24:53.697 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:24:53.695+0000 7faf91824d80 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2020-06-04T08:24:53.697 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:24:53.695+0000 7faf91824d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-06-04T08:24:53.698 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:24:53.695+0000 7faf91824d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-06-04T08:24:53.698 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:24:53.695+0000 7faf91824d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2020-06-04T08:24:53.699 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:24:53.695+0000 7faf91824d80 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2020-06-04T08:24:55.638 INFO:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2020-06-04T08:24:55.694 INFO:tasks.ceph:Reading keys from all nodes... 2020-06-04T08:24:55.695 INFO:teuthology.orchestra.run.smithi196:> mktemp 2020-06-04T08:24:55.740 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.gLrSgexl4T 2020-06-04T08:24:55.740 INFO:teuthology.orchestra.run.smithi196:> sudo cp /var/lib/ceph/mgr/c2-x/keyring /tmp/tmp.gLrSgexl4T 2020-06-04T08:24:55.807 INFO:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.gLrSgexl4T 2020-06-04T08:24:55.918 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.gLrSgexl4T is 56B 2020-06-04T08:24:55.930 INFO:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.gLrSgexl4T 2020-06-04T08:24:55.936 INFO:teuthology.orchestra.run.smithi196:> mktemp 2020-06-04T08:24:55.985 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.duJRljSzsd 2020-06-04T08:24:55.985 INFO:teuthology.orchestra.run.smithi196:> sudo cp /var/lib/ceph/osd/c2-0/keyring /tmp/tmp.duJRljSzsd 2020-06-04T08:24:56.086 INFO:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.duJRljSzsd 2020-06-04T08:24:56.242 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.duJRljSzsd is 56B 2020-06-04T08:24:56.323 INFO:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.duJRljSzsd 2020-06-04T08:24:56.344 INFO:teuthology.orchestra.run.smithi196:> mktemp 2020-06-04T08:24:56.413 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.5jzeMgtzMb 2020-06-04T08:24:56.414 INFO:teuthology.orchestra.run.smithi196:> sudo cp /var/lib/ceph/osd/c2-1/keyring /tmp/tmp.5jzeMgtzMb 2020-06-04T08:24:56.489 INFO:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.5jzeMgtzMb 2020-06-04T08:24:56.599 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.5jzeMgtzMb is 56B 2020-06-04T08:24:56.612 INFO:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.5jzeMgtzMb 2020-06-04T08:24:56.618 INFO:teuthology.orchestra.run.smithi196:> mktemp 2020-06-04T08:24:56.668 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.vVgwYePVxN 2020-06-04T08:24:56.669 INFO:teuthology.orchestra.run.smithi196:> sudo cp /var/lib/ceph/osd/c2-2/keyring /tmp/tmp.vVgwYePVxN 2020-06-04T08:24:56.726 INFO:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.vVgwYePVxN 2020-06-04T08:24:56.844 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.vVgwYePVxN is 56B 2020-06-04T08:24:56.859 INFO:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.vVgwYePVxN 2020-06-04T08:24:56.919 DEBUG:teuthology.orchestra.remote:smithi196:/etc/ceph/c2.client.0.keyring is 59B 2020-06-04T08:24:56.941 DEBUG:teuthology.orchestra.remote:smithi196:/etc/ceph/c2.client.1.keyring is 59B 2020-06-04T08:24:56.952 INFO:tasks.ceph:Adding keys to all mons... 2020-06-04T08:24:56.953 INFO:teuthology.orchestra.run.smithi196:> sudo tee -a /etc/ceph/c2.keyring 2020-06-04T08:24:56.966 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr.x] 2020-06-04T08:24:56.967 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDPr9heeXoSEBAA7iUg6PFHvS6uyYCrOlR7jQ== 2020-06-04T08:24:56.967 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.0] 2020-06-04T08:24:56.967 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDRr9heY7K8IxAAnGOMGsLpdBchWmxe07JBzA== 2020-06-04T08:24:56.967 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.1] 2020-06-04T08:24:56.968 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDTr9heAABlJRAAO0eKKc7IHfd2qkJvgjLgsQ== 2020-06-04T08:24:56.968 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.2] 2020-06-04T08:24:56.968 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDVr9hejzN6KRAAT/nQOyDehWd02f/7jvoybA== 2020-06-04T08:24:56.968 INFO:teuthology.orchestra.run.smithi196.stdout:[client.0] 2020-06-04T08:24:56.968 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDPr9he1Zj4FBAAVY8TdA7ctB04CbvZaeozBA== 2020-06-04T08:24:56.969 INFO:teuthology.orchestra.run.smithi196.stdout:[client.1] 2020-06-04T08:24:56.969 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDPr9heR9s1GhAAzX/jqytx/wRkOt3CSAD5pg== 2020-06-04T08:24:56.970 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2020-06-04T08:24:57.055 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-04T08:24:57.143 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-04T08:24:57.225 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2020-06-04T08:24:57.327 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-06-04T08:24:57.416 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2020-06-04T08:24:57.507 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-06-04T08:24:57.507 INFO:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2020-06-04T08:24:57.565 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2020-06-04T08:24:57.659 INFO:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2020-06-04T08:24:57.678 INFO:teuthology.orchestra.run.smithi196:> rm -- /home/ubuntu/cephtest/c2.monmap 2020-06-04T08:24:57.725 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2020-06-04T08:24:57.726 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2020-06-04T08:24:57.726 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2020-06-04T08:24:57.769 INFO:tasks.ceph.c2.mon.a:Started 2020-06-04T08:24:57.769 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2020-06-04T08:24:57.770 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2020-06-04T08:24:57.770 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2020-06-04T08:24:57.773 INFO:tasks.ceph.c2.mgr.x:Started 2020-06-04T08:24:57.774 INFO:tasks.ceph:Setting crush tunables to default 2020-06-04T08:24:57.774 INFO:teuthology.orchestra.run.smithi196:> sudo ceph --cluster c2 osd crush tunables default 2020-06-04T08:24:58.113 INFO:teuthology.orchestra.run.smithi196.stderr:adjusted tunables profile to default 2020-06-04T08:24:58.124 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2020-06-04T08:24:58.125 INFO:teuthology.orchestra.run.smithi196:> mktemp 2020-06-04T08:24:58.129 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.Fhmyn0dbOd 2020-06-04T08:24:58.130 INFO:teuthology.orchestra.run.smithi196:> sudo cp /var/lib/ceph/osd/c2-0/fsid /tmp/tmp.Fhmyn0dbOd 2020-06-04T08:24:58.186 INFO:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.Fhmyn0dbOd 2020-06-04T08:24:58.290 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.Fhmyn0dbOd is 37B 2020-06-04T08:24:58.300 INFO:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.Fhmyn0dbOd 2020-06-04T08:24:58.308 INFO:teuthology.orchestra.run.smithi196:> mktemp 2020-06-04T08:24:58.356 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.NN1p6NGDvr 2020-06-04T08:24:58.357 INFO:teuthology.orchestra.run.smithi196:> sudo cp /var/lib/ceph/osd/c2-1/fsid /tmp/tmp.NN1p6NGDvr 2020-06-04T08:24:58.413 INFO:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.NN1p6NGDvr 2020-06-04T08:24:58.518 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.NN1p6NGDvr is 37B 2020-06-04T08:24:58.557 INFO:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.NN1p6NGDvr 2020-06-04T08:24:58.566 INFO:teuthology.orchestra.run.smithi196:> mktemp 2020-06-04T08:24:58.613 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.7nZhD1vBb7 2020-06-04T08:24:58.613 INFO:teuthology.orchestra.run.smithi196:> sudo cp /var/lib/ceph/osd/c2-2/fsid /tmp/tmp.7nZhD1vBb7 2020-06-04T08:24:58.669 INFO:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.7nZhD1vBb7 2020-06-04T08:24:58.777 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.7nZhD1vBb7 is 37B 2020-06-04T08:24:58.828 INFO:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.7nZhD1vBb7 2020-06-04T08:24:58.834 INFO:teuthology.orchestra.run.smithi196:> sudo ceph --cluster c2 osd new faffec15-dd3f-440b-8013-415f681492ae 0 2020-06-04T08:24:59.078 INFO:teuthology.orchestra.run.smithi196.stdout:0 2020-06-04T08:24:59.090 INFO:teuthology.orchestra.run.smithi196:> sudo ceph --cluster c2 osd new ec1ac180-603b-4fac-8577-514b08ed7be7 1 2020-06-04T08:24:59.289 INFO:teuthology.orchestra.run.smithi196.stdout:1 2020-06-04T08:24:59.300 INFO:teuthology.orchestra.run.smithi196:> sudo ceph --cluster c2 osd new c71bd269-cfd9-46f2-b7be-986545bbae28 2 2020-06-04T08:24:59.504 INFO:teuthology.orchestra.run.smithi196.stdout:2 2020-06-04T08:24:59.516 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2020-06-04T08:24:59.516 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2020-06-04T08:24:59.518 INFO:tasks.ceph.c2.osd.0:Started 2020-06-04T08:24:59.519 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2020-06-04T08:24:59.519 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2020-06-04T08:24:59.526 INFO:tasks.ceph.c2.osd.1:Started 2020-06-04T08:24:59.527 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2020-06-04T08:24:59.527 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2020-06-04T08:24:59.532 INFO:tasks.ceph.c2.osd.2:Started 2020-06-04T08:24:59.532 INFO:tasks.ceph:Waiting for OSDs to come up 2020-06-04T08:24:59.835 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-06-04T08:24:59.883 INFO:tasks.ceph.c2.osd.0.smithi196.stderr:2020-06-04T08:24:59.879+0000 7f45e787ed80 -1 Falling back to public interface 2020-06-04T08:25:00.007 INFO:teuthology.misc.health.smithi196.stdout: 2020-06-04T08:25:00.007 INFO:teuthology.misc.health.smithi196.stdout:{"epoch":5,"fsid":"f2c8e8e3-2947-4b15-b4d7-3d00a5ceca6c","created":"2020-06-04T08:24:57.910076+0000","modified":"2020-06-04T08:24:59.501032+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"faffec15-dd3f-440b-8013-415f681492ae","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ec1ac180-603b-4fac-8577-514b08ed7be7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c71bd269-cfd9-46f2-b7be-986545bbae28","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-04T08:25:00.018 DEBUG:teuthology.misc:0 of 3 OSDs are up 2020-06-04T08:25:00.142 INFO:tasks.ceph.c2.osd.1.smithi196.stderr:2020-06-04T08:25:00.139+0000 7fdbf9ae9d80 -1 Falling back to public interface 2020-06-04T08:25:00.165 INFO:tasks.ceph.c2.osd.2.smithi196.stderr:2020-06-04T08:25:00.163+0000 7fc81285bd80 -1 Falling back to public interface 2020-06-04T08:25:00.520 INFO:tasks.ceph.c2.osd.1.smithi196.stderr:2020-06-04T08:25:00.519+0000 7fdbf9ae9d80 -1 osd.1 0 log_to_monitors {default=true} 2020-06-04T08:25:00.522 INFO:tasks.ceph.c2.osd.0.smithi196.stderr:2020-06-04T08:25:00.519+0000 7f45e787ed80 -1 osd.0 0 log_to_monitors {default=true} 2020-06-04T08:25:00.798 INFO:tasks.ceph.c2.osd.2.smithi196.stderr:2020-06-04T08:25:00.795+0000 7fc81285bd80 -1 osd.2 0 log_to_monitors {default=true} 2020-06-04T08:25:01.947 INFO:tasks.ceph.c2.osd.0.smithi196.stderr:2020-06-04T08:25:01.943+0000 7f45ded49700 -1 osd.0 0 waiting for initial osdmap 2020-06-04T08:25:01.948 INFO:tasks.ceph.c2.osd.1.smithi196.stderr:2020-06-04T08:25:01.943+0000 7fdbf0fb4700 -1 osd.1 0 waiting for initial osdmap 2020-06-04T08:25:01.948 INFO:tasks.ceph.c2.osd.2.smithi196.stderr:2020-06-04T08:25:01.943+0000 7fc80b529700 -1 osd.2 0 waiting for initial osdmap 2020-06-04T08:25:06.322 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2020-06-04T08:25:06.580 INFO:teuthology.misc.health.smithi196.stdout: 2020-06-04T08:25:06.580 INFO:teuthology.misc.health.smithi196.stdout:{"epoch":10,"fsid":"f2c8e8e3-2947-4b15-b4d7-3d00a5ceca6c","created":"2020-06-04T08:24:57.910076+0000","modified":"2020-06-04T08:25:04.957715+0000","last_up_change":"2020-06-04T08:25:02.933769+0000","last_in_change":"2020-06-04T08:25:02.933769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-04T08:25:01.588610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"faffec15-dd3f-440b-8013-415f681492ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":13673},{"type":"v1","addr":"172.21.15.196:6801","nonce":13673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":13673},{"type":"v1","addr":"172.21.15.196:6803","nonce":13673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":13673},{"type":"v1","addr":"172.21.15.196:6807","nonce":13673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":13673},{"type":"v1","addr":"172.21.15.196:6805","nonce":13673}]},"public_addr":"172.21.15.196:6801/13673","cluster_addr":"172.21.15.196:6803/13673","heartbeat_back_addr":"172.21.15.196:6807/13673","heartbeat_front_addr":"172.21.15.196:6805/13673","state":["exists","up"]},{"osd":1,"uuid":"ec1ac180-603b-4fac-8577-514b08ed7be7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":13674},{"type":"v1","addr":"172.21.15.196:6809","nonce":13674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":13674},{"type":"v1","addr":"172.21.15.196:6811","nonce":13674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":13674},{"type":"v1","addr":"172.21.15.196:6815","nonce":13674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":13674},{"type":"v1","addr":"172.21.15.196:6813","nonce":13674}]},"public_addr":"172.21.15.196:6809/13674","cluster_addr":"172.21.15.196:6811/13674","heartbeat_back_addr":"172.21.15.196:6815/13674","heartbeat_front_addr":"172.21.15.196:6813/13674","state":["exists","up"]},{"osd":2,"uuid":"c71bd269-cfd9-46f2-b7be-986545bbae28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":13677},{"type":"v1","addr":"172.21.15.196:6817","nonce":13677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":13677},{"type":"v1","addr":"172.21.15.196:6819","nonce":13677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":13677},{"type":"v1","addr":"172.21.15.196:6823","nonce":13677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":13677},{"type":"v1","addr":"172.21.15.196:6821","nonce":13677}]},"public_addr":"172.21.15.196:6817/13677","cluster_addr":"172.21.15.196:6819/13677","heartbeat_back_addr":"172.21.15.196:6823/13677","heartbeat_front_addr":"172.21.15.196:6821/13677","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:25:01.518873+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:25:01.472951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:25:01.778102+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-04T08:25:06.592 DEBUG:teuthology.misc:3 of 3 OSDs are up 2020-06-04T08:25:06.592 INFO:tasks.ceph:Creating RBD pool 2020-06-04T08:25:06.592 INFO:teuthology.orchestra.run.smithi196:> sudo ceph --cluster c2 osd pool create rbd 8 2020-06-04T08:25:06.994 INFO:teuthology.orchestra.run.smithi196.stderr:pool 'rbd' created 2020-06-04T08:25:07.004 INFO:teuthology.orchestra.run.smithi196:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2020-06-04T08:25:08.000 INFO:teuthology.orchestra.run.smithi196.stderr:enabled application 'rbd' on pool 'rbd' 2020-06-04T08:25:08.012 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2020-06-04T08:25:08.013 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-06-04T08:25:08.014 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-06-04T08:25:08.268 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:25:08.268 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":12,"fsid":"f2c8e8e3-2947-4b15-b4d7-3d00a5ceca6c","created":"2020-06-04T08:24:57.910076+0000","modified":"2020-06-04T08:25:07.978992+0000","last_up_change":"2020-06-04T08:25:02.933769+0000","last_in_change":"2020-06-04T08:25:02.933769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-04T08:25:01.588610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-06-04T08:25:06.845071+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"faffec15-dd3f-440b-8013-415f681492ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":13673},{"type":"v1","addr":"172.21.15.196:6801","nonce":13673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":13673},{"type":"v1","addr":"172.21.15.196:6803","nonce":13673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":13673},{"type":"v1","addr":"172.21.15.196:6807","nonce":13673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":13673},{"type":"v1","addr":"172.21.15.196:6805","nonce":13673}]},"public_addr":"172.21.15.196:6801/13673","cluster_addr":"172.21.15.196:6803/13673","heartbeat_back_addr":"172.21.15.196:6807/13673","heartbeat_front_addr":"172.21.15.196:6805/13673","state":["exists","up"]},{"osd":1,"uuid":"ec1ac180-603b-4fac-8577-514b08ed7be7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":13674},{"type":"v1","addr":"172.21.15.196:6809","nonce":13674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":13674},{"type":"v1","addr":"172.21.15.196:6811","nonce":13674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":13674},{"type":"v1","addr":"172.21.15.196:6815","nonce":13674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":13674},{"type":"v1","addr":"172.21.15.196:6813","nonce":13674}]},"public_addr":"172.21.15.196:6809/13674","cluster_addr":"172.21.15.196:6811/13674","heartbeat_back_addr":"172.21.15.196:6815/13674","heartbeat_front_addr":"172.21.15.196:6813/13674","state":["exists","up"]},{"osd":2,"uuid":"c71bd269-cfd9-46f2-b7be-986545bbae28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":13677},{"type":"v1","addr":"172.21.15.196:6817","nonce":13677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":13677},{"type":"v1","addr":"172.21.15.196:6819","nonce":13677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":13677},{"type":"v1","addr":"172.21.15.196:6823","nonce":13677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":13677},{"type":"v1","addr":"172.21.15.196:6821","nonce":13677}]},"public_addr":"172.21.15.196:6817/13677","cluster_addr":"172.21.15.196:6819/13677","heartbeat_back_addr":"172.21.15.196:6823/13677","heartbeat_front_addr":"172.21.15.196:6821/13677","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:25:01.518873+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:25:01.472951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:25:01.778102+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-04T08:25:08.281 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2020-06-04T08:25:01.588610+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2020-06-04T08:25:06.845071+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2020-06-04T08:25:08.282 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get device_health_metrics pg_num 2020-06-04T08:25:08.548 INFO:teuthology.orchestra.run.smithi196.stdout:pg_num: 1 2020-06-04T08:25:08.558 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get rbd pg_num 2020-06-04T08:25:08.833 INFO:teuthology.orchestra.run.smithi196.stdout:pg_num: 8 2020-06-04T08:25:08.846 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2020-06-04T08:25:08.846 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2020-06-04T08:25:08.847 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early mgr dump --format=json 2020-06-04T08:25:09.147 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:25:09.170 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":4,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":13471},{"type":"v1","addr":"172.21.15.196:6825","nonce":13471}]},"active_addr":"172.21.15.196:6825/13471","active_change":"2020-06-04T08:25:01.561232+0000","active_mgr_features":4540138292837744639,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"ceph/ceph-grafana:latest","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","osd_support","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","osd_support","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.196:0","nonce":1156678485}]},{"addrvec":[{"type":"v2","addr":"172.21.15.196:0","nonce":1744186326}]},{"addrvec":[{"type":"v2","addr":"172.21.15.196:0","nonce":3137803349}]}]}} 2020-06-04T08:25:09.174 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2020-06-04T08:25:09.174 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2020-06-04T08:25:09.175 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-06-04T08:25:09.435 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:25:09.435 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":13,"fsid":"f2c8e8e3-2947-4b15-b4d7-3d00a5ceca6c","created":"2020-06-04T08:24:57.910076+0000","modified":"2020-06-04T08:25:08.983750+0000","last_up_change":"2020-06-04T08:25:02.933769+0000","last_in_change":"2020-06-04T08:25:02.933769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-04T08:25:01.588610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-06-04T08:25:06.845071+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"faffec15-dd3f-440b-8013-415f681492ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":13673},{"type":"v1","addr":"172.21.15.196:6801","nonce":13673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":13673},{"type":"v1","addr":"172.21.15.196:6803","nonce":13673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":13673},{"type":"v1","addr":"172.21.15.196:6807","nonce":13673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":13673},{"type":"v1","addr":"172.21.15.196:6805","nonce":13673}]},"public_addr":"172.21.15.196:6801/13673","cluster_addr":"172.21.15.196:6803/13673","heartbeat_back_addr":"172.21.15.196:6807/13673","heartbeat_front_addr":"172.21.15.196:6805/13673","state":["exists","up"]},{"osd":1,"uuid":"ec1ac180-603b-4fac-8577-514b08ed7be7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":13674},{"type":"v1","addr":"172.21.15.196:6809","nonce":13674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":13674},{"type":"v1","addr":"172.21.15.196:6811","nonce":13674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":13674},{"type":"v1","addr":"172.21.15.196:6815","nonce":13674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":13674},{"type":"v1","addr":"172.21.15.196:6813","nonce":13674}]},"public_addr":"172.21.15.196:6809/13674","cluster_addr":"172.21.15.196:6811/13674","heartbeat_back_addr":"172.21.15.196:6815/13674","heartbeat_front_addr":"172.21.15.196:6813/13674","state":["exists","up"]},{"osd":2,"uuid":"c71bd269-cfd9-46f2-b7be-986545bbae28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":13677},{"type":"v1","addr":"172.21.15.196:6817","nonce":13677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":13677},{"type":"v1","addr":"172.21.15.196:6819","nonce":13677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":13677},{"type":"v1","addr":"172.21.15.196:6823","nonce":13677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":13677},{"type":"v1","addr":"172.21.15.196:6821","nonce":13677}]},"public_addr":"172.21.15.196:6817/13677","cluster_addr":"172.21.15.196:6819/13677","heartbeat_back_addr":"172.21.15.196:6823/13677","heartbeat_front_addr":"172.21.15.196:6821/13677","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:25:01.518873+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:25:01.472951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:25:01.778102+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-04T08:25:09.448 INFO:tasks.ceph.ceph_manager.c2:all up! 2020-06-04T08:25:09.448 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2020-06-04T08:25:09.715 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:25:09.715 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":13,"fsid":"f2c8e8e3-2947-4b15-b4d7-3d00a5ceca6c","created":"2020-06-04T08:24:57.910076+0000","modified":"2020-06-04T08:25:08.983750+0000","last_up_change":"2020-06-04T08:25:02.933769+0000","last_in_change":"2020-06-04T08:25:02.933769+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2020-06-04T08:25:01.588610+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2020-06-04T08:25:06.845071+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"faffec15-dd3f-440b-8013-415f681492ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":13673},{"type":"v1","addr":"172.21.15.196:6801","nonce":13673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":13673},{"type":"v1","addr":"172.21.15.196:6803","nonce":13673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":13673},{"type":"v1","addr":"172.21.15.196:6807","nonce":13673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":13673},{"type":"v1","addr":"172.21.15.196:6805","nonce":13673}]},"public_addr":"172.21.15.196:6801/13673","cluster_addr":"172.21.15.196:6803/13673","heartbeat_back_addr":"172.21.15.196:6807/13673","heartbeat_front_addr":"172.21.15.196:6805/13673","state":["exists","up"]},{"osd":1,"uuid":"ec1ac180-603b-4fac-8577-514b08ed7be7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":13674},{"type":"v1","addr":"172.21.15.196:6809","nonce":13674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":13674},{"type":"v1","addr":"172.21.15.196:6811","nonce":13674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":13674},{"type":"v1","addr":"172.21.15.196:6815","nonce":13674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":13674},{"type":"v1","addr":"172.21.15.196:6813","nonce":13674}]},"public_addr":"172.21.15.196:6809/13674","cluster_addr":"172.21.15.196:6811/13674","heartbeat_back_addr":"172.21.15.196:6815/13674","heartbeat_front_addr":"172.21.15.196:6813/13674","state":["exists","up"]},{"osd":2,"uuid":"c71bd269-cfd9-46f2-b7be-986545bbae28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":13677},{"type":"v1","addr":"172.21.15.196:6817","nonce":13677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":13677},{"type":"v1","addr":"172.21.15.196:6819","nonce":13677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":13677},{"type":"v1","addr":"172.21.15.196:6823","nonce":13677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":13677},{"type":"v1","addr":"172.21.15.196:6821","nonce":13677}]},"public_addr":"172.21.15.196:6817/13677","cluster_addr":"172.21.15.196:6819/13677","heartbeat_back_addr":"172.21.15.196:6823/13677","heartbeat_front_addr":"172.21.15.196:6821/13677","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:25:01.518873+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:25:01.472951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292837744639,"old_weight":0,"last_purged_snaps_scrub":"2020-06-04T08:25:01.778102+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-06-04T08:25:09.728 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.0 flush_pg_stats 2020-06-04T08:25:09.885 INFO:teuthology.orchestra.run.smithi196.stdout:34359738371 2020-06-04T08:25:09.886 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.1 flush_pg_stats 2020-06-04T08:25:10.040 INFO:teuthology.orchestra.run.smithi196.stdout:34359738371 2020-06-04T08:25:10.040 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.2 flush_pg_stats 2020-06-04T08:25:10.195 INFO:teuthology.orchestra.run.smithi196.stdout:34359738372 2020-06-04T08:25:10.196 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2020-06-04T08:25:10.462 INFO:teuthology.orchestra.run.smithi196.stdout:34359738370 2020-06-04T08:25:10.476 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2020-06-04T08:25:11.477 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2020-06-04T08:25:11.749 INFO:teuthology.orchestra.run.smithi196.stdout:34359738370 2020-06-04T08:25:11.763 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2020-06-04T08:25:12.764 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2020-06-04T08:25:13.038 INFO:teuthology.orchestra.run.smithi196.stdout:34359738371 2020-06-04T08:25:13.051 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.0 2020-06-04T08:25:13.052 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.1 2020-06-04T08:25:13.314 INFO:teuthology.orchestra.run.smithi196.stdout:34359738371 2020-06-04T08:25:13.326 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.1 2020-06-04T08:25:13.326 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.2 2020-06-04T08:25:13.581 INFO:teuthology.orchestra.run.smithi196.stdout:34359738372 2020-06-04T08:25:13.592 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.2 2020-06-04T08:25:13.592 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2020-06-04T08:25:13.593 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2020-06-04T08:25:13.858 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:25:13.859 INFO:teuthology.orchestra.run.smithi196.stderr:dumped all 2020-06-04T08:25:13.872 INFO:teuthology.orchestra.run.smithi196.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-06-04T08:25:13.569496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146080,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426720,"statfs":{"total":322122547200,"available":318900961280,"internally_reserved":3221225472,"allocated":335872,"data_stored":95895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.583623"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:25:08.000608+0000","last_change":"2020-06-04T08:25:08.000156+0000","last_active":"2020-06-04T08:25:08.000608+0000","last_peered":"2020-06-04T08:25:08.000608+0000","last_clean":"2020-06-04T08:25:08.000608+0000","last_became_active":"2020-06-04T08:25:07.999552+0000","last_became_peered":"2020-06-04T08:25:07.999552+0000","last_unstale":"2020-06-04T08:25:08.000608+0000","last_undegraded":"2020-06-04T08:25:08.000608+0000","last_fullsized":"2020-06-04T08:25:08.000608+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_clean_scrub_stamp":"2020-06-04T08:25:06.970748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:25:08.000455+0000","last_change":"2020-06-04T08:25:07.999979+0000","last_active":"2020-06-04T08:25:08.000455+0000","last_peered":"2020-06-04T08:25:08.000455+0000","last_clean":"2020-06-04T08:25:08.000455+0000","last_became_active":"2020-06-04T08:25:07.997080+0000","last_became_peered":"2020-06-04T08:25:07.997080+0000","last_unstale":"2020-06-04T08:25:08.000455+0000","last_undegraded":"2020-06-04T08:25:08.000455+0000","last_fullsized":"2020-06-04T08:25:08.000455+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_clean_scrub_stamp":"2020-06-04T08:25:06.970748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:25:08.000294+0000","last_change":"2020-06-04T08:25:07.999378+0000","last_active":"2020-06-04T08:25:08.000294+0000","last_peered":"2020-06-04T08:25:08.000294+0000","last_clean":"2020-06-04T08:25:08.000294+0000","last_became_active":"2020-06-04T08:25:07.997911+0000","last_became_peered":"2020-06-04T08:25:07.997911+0000","last_unstale":"2020-06-04T08:25:08.000294+0000","last_undegraded":"2020-06-04T08:25:08.000294+0000","last_fullsized":"2020-06-04T08:25:08.000294+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_clean_scrub_stamp":"2020-06-04T08:25:06.970748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:25:07.997861+0000","last_change":"2020-06-04T08:25:07.997217+0000","last_active":"2020-06-04T08:25:07.997861+0000","last_peered":"2020-06-04T08:25:07.997861+0000","last_clean":"2020-06-04T08:25:07.997861+0000","last_became_active":"2020-06-04T08:25:07.996716+0000","last_became_peered":"2020-06-04T08:25:07.996716+0000","last_unstale":"2020-06-04T08:25:07.997861+0000","last_undegraded":"2020-06-04T08:25:07.997861+0000","last_fullsized":"2020-06-04T08:25:07.997861+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_clean_scrub_stamp":"2020-06-04T08:25:06.970748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:25:07.994596+0000","last_change":"2020-06-04T08:25:04.443278+0000","last_active":"2020-06-04T08:25:07.994596+0000","last_peered":"2020-06-04T08:25:07.994596+0000","last_clean":"2020-06-04T08:25:07.994596+0000","last_became_active":"2020-06-04T08:25:04.441655+0000","last_became_peered":"2020-06-04T08:25:04.441655+0000","last_unstale":"2020-06-04T08:25:07.994596+0000","last_undegraded":"2020-06-04T08:25:07.994596+0000","last_fullsized":"2020-06-04T08:25:07.994596+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:25:01.924263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:25:01.924263+0000","last_clean_scrub_stamp":"2020-06-04T08:25:01.924263+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:25:08.000167+0000","last_change":"2020-06-04T08:25:07.999533+0000","last_active":"2020-06-04T08:25:08.000167+0000","last_peered":"2020-06-04T08:25:08.000167+0000","last_clean":"2020-06-04T08:25:08.000167+0000","last_became_active":"2020-06-04T08:25:07.999132+0000","last_became_peered":"2020-06-04T08:25:07.999132+0000","last_unstale":"2020-06-04T08:25:08.000167+0000","last_undegraded":"2020-06-04T08:25:08.000167+0000","last_fullsized":"2020-06-04T08:25:08.000167+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_clean_scrub_stamp":"2020-06-04T08:25:06.970748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:25:07.997849+0000","last_change":"2020-06-04T08:25:07.997158+0000","last_active":"2020-06-04T08:25:07.997849+0000","last_peered":"2020-06-04T08:25:07.997849+0000","last_clean":"2020-06-04T08:25:07.997849+0000","last_became_active":"2020-06-04T08:25:07.996598+0000","last_became_peered":"2020-06-04T08:25:07.996598+0000","last_unstale":"2020-06-04T08:25:07.997849+0000","last_undegraded":"2020-06-04T08:25:07.997849+0000","last_fullsized":"2020-06-04T08:25:07.997849+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_clean_scrub_stamp":"2020-06-04T08:25:06.970748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:25:07.997530+0000","last_change":"2020-06-04T08:25:07.996803+0000","last_active":"2020-06-04T08:25:07.997530+0000","last_peered":"2020-06-04T08:25:07.997530+0000","last_clean":"2020-06-04T08:25:07.997530+0000","last_became_active":"2020-06-04T08:25:07.996373+0000","last_became_peered":"2020-06-04T08:25:07.996373+0000","last_unstale":"2020-06-04T08:25:07.997530+0000","last_undegraded":"2020-06-04T08:25:07.997530+0000","last_fullsized":"2020-06-04T08:25:07.997530+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_clean_scrub_stamp":"2020-06-04T08:25:06.970748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:25:07.999190+0000","last_change":"2020-06-04T08:25:07.998358+0000","last_active":"2020-06-04T08:25:07.999190+0000","last_peered":"2020-06-04T08:25:07.999190+0000","last_clean":"2020-06-04T08:25:07.999190+0000","last_became_active":"2020-06-04T08:25:07.997527+0000","last_became_peered":"2020-06-04T08:25:07.997527+0000","last_unstale":"2020-06-04T08:25:07.999190+0000","last_undegraded":"2020-06-04T08:25:07.999190+0000","last_fullsized":"2020-06-04T08:25:07.999190+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_clean_scrub_stamp":"2020-06-04T08:25:06.970748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1879999999999999}]}]}]}} 2020-06-04T08:25:13.875 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2020-06-04T08:25:14.130 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:25:14.132 INFO:teuthology.orchestra.run.smithi196.stderr:dumped all 2020-06-04T08:25:14.142 INFO:teuthology.orchestra.run.smithi196.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2020-06-04T08:25:13.569496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146080,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426720,"statfs":{"total":322122547200,"available":318900961280,"internally_reserved":3221225472,"allocated":335872,"data_stored":95895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.583623"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:25:08.000608+0000","last_change":"2020-06-04T08:25:08.000156+0000","last_active":"2020-06-04T08:25:08.000608+0000","last_peered":"2020-06-04T08:25:08.000608+0000","last_clean":"2020-06-04T08:25:08.000608+0000","last_became_active":"2020-06-04T08:25:07.999552+0000","last_became_peered":"2020-06-04T08:25:07.999552+0000","last_unstale":"2020-06-04T08:25:08.000608+0000","last_undegraded":"2020-06-04T08:25:08.000608+0000","last_fullsized":"2020-06-04T08:25:08.000608+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_clean_scrub_stamp":"2020-06-04T08:25:06.970748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:25:08.000455+0000","last_change":"2020-06-04T08:25:07.999979+0000","last_active":"2020-06-04T08:25:08.000455+0000","last_peered":"2020-06-04T08:25:08.000455+0000","last_clean":"2020-06-04T08:25:08.000455+0000","last_became_active":"2020-06-04T08:25:07.997080+0000","last_became_peered":"2020-06-04T08:25:07.997080+0000","last_unstale":"2020-06-04T08:25:08.000455+0000","last_undegraded":"2020-06-04T08:25:08.000455+0000","last_fullsized":"2020-06-04T08:25:08.000455+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_clean_scrub_stamp":"2020-06-04T08:25:06.970748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:25:08.000294+0000","last_change":"2020-06-04T08:25:07.999378+0000","last_active":"2020-06-04T08:25:08.000294+0000","last_peered":"2020-06-04T08:25:08.000294+0000","last_clean":"2020-06-04T08:25:08.000294+0000","last_became_active":"2020-06-04T08:25:07.997911+0000","last_became_peered":"2020-06-04T08:25:07.997911+0000","last_unstale":"2020-06-04T08:25:08.000294+0000","last_undegraded":"2020-06-04T08:25:08.000294+0000","last_fullsized":"2020-06-04T08:25:08.000294+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_clean_scrub_stamp":"2020-06-04T08:25:06.970748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:25:07.997861+0000","last_change":"2020-06-04T08:25:07.997217+0000","last_active":"2020-06-04T08:25:07.997861+0000","last_peered":"2020-06-04T08:25:07.997861+0000","last_clean":"2020-06-04T08:25:07.997861+0000","last_became_active":"2020-06-04T08:25:07.996716+0000","last_became_peered":"2020-06-04T08:25:07.996716+0000","last_unstale":"2020-06-04T08:25:07.997861+0000","last_undegraded":"2020-06-04T08:25:07.997861+0000","last_fullsized":"2020-06-04T08:25:07.997861+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_clean_scrub_stamp":"2020-06-04T08:25:06.970748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"12","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:25:07.994596+0000","last_change":"2020-06-04T08:25:04.443278+0000","last_active":"2020-06-04T08:25:07.994596+0000","last_peered":"2020-06-04T08:25:07.994596+0000","last_clean":"2020-06-04T08:25:07.994596+0000","last_became_active":"2020-06-04T08:25:04.441655+0000","last_became_peered":"2020-06-04T08:25:04.441655+0000","last_unstale":"2020-06-04T08:25:07.994596+0000","last_undegraded":"2020-06-04T08:25:07.994596+0000","last_fullsized":"2020-06-04T08:25:07.994596+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:25:01.924263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:25:01.924263+0000","last_clean_scrub_stamp":"2020-06-04T08:25:01.924263+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:25:08.000167+0000","last_change":"2020-06-04T08:25:07.999533+0000","last_active":"2020-06-04T08:25:08.000167+0000","last_peered":"2020-06-04T08:25:08.000167+0000","last_clean":"2020-06-04T08:25:08.000167+0000","last_became_active":"2020-06-04T08:25:07.999132+0000","last_became_peered":"2020-06-04T08:25:07.999132+0000","last_unstale":"2020-06-04T08:25:08.000167+0000","last_undegraded":"2020-06-04T08:25:08.000167+0000","last_fullsized":"2020-06-04T08:25:08.000167+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_clean_scrub_stamp":"2020-06-04T08:25:06.970748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:25:07.997849+0000","last_change":"2020-06-04T08:25:07.997158+0000","last_active":"2020-06-04T08:25:07.997849+0000","last_peered":"2020-06-04T08:25:07.997849+0000","last_clean":"2020-06-04T08:25:07.997849+0000","last_became_active":"2020-06-04T08:25:07.996598+0000","last_became_peered":"2020-06-04T08:25:07.996598+0000","last_unstale":"2020-06-04T08:25:07.997849+0000","last_undegraded":"2020-06-04T08:25:07.997849+0000","last_fullsized":"2020-06-04T08:25:07.997849+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_clean_scrub_stamp":"2020-06-04T08:25:06.970748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:25:07.997530+0000","last_change":"2020-06-04T08:25:07.996803+0000","last_active":"2020-06-04T08:25:07.997530+0000","last_peered":"2020-06-04T08:25:07.997530+0000","last_clean":"2020-06-04T08:25:07.997530+0000","last_became_active":"2020-06-04T08:25:07.996373+0000","last_became_peered":"2020-06-04T08:25:07.996373+0000","last_unstale":"2020-06-04T08:25:07.997530+0000","last_undegraded":"2020-06-04T08:25:07.997530+0000","last_fullsized":"2020-06-04T08:25:07.997530+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_clean_scrub_stamp":"2020-06-04T08:25:06.970748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2020-06-04T08:25:07.999190+0000","last_change":"2020-06-04T08:25:07.998358+0000","last_active":"2020-06-04T08:25:07.999190+0000","last_peered":"2020-06-04T08:25:07.999190+0000","last_clean":"2020-06-04T08:25:07.999190+0000","last_became_active":"2020-06-04T08:25:07.997527+0000","last_became_peered":"2020-06-04T08:25:07.997527+0000","last_unstale":"2020-06-04T08:25:07.999190+0000","last_undegraded":"2020-06-04T08:25:07.999190+0000","last_fullsized":"2020-06-04T08:25:07.999190+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-06-04T08:25:06.970748+0000","last_clean_scrub_stamp":"2020-06-04T08:25:06.970748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1879999999999999}]}]}]}} 2020-06-04T08:25:14.144 INFO:tasks.ceph.ceph_manager.c2:clean! 2020-06-04T08:25:14.144 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2020-06-04T08:25:14.145 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2020-06-04T08:25:14.145 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early health --format=json 2020-06-04T08:25:14.454 INFO:teuthology.orchestra.run.smithi196.stdout: 2020-06-04T08:25:14.467 INFO:teuthology.orchestra.run.smithi196.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-06-04T08:25:14.468 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2020-06-04T08:25:14.469 INFO:teuthology.run_tasks:Running task rgw... 2020-06-04T08:25:14.542 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2020-06-04T08:25:14.543 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2020-06-04T08:25:14.544 INFO:tasks.rgw:Creating data pools 2020-06-04T08:25:14.545 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2020-06-04T08:25:14.546 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-06-04T08:25:15.548 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'default.rgw.buckets.data' created 2020-06-04T08:25:15.566 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-06-04T08:25:16.576 INFO:teuthology.orchestra.run.smithi060.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-06-04T08:25:16.596 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-06-04T08:25:17.574 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'default.rgw.buckets.index' created 2020-06-04T08:25:17.594 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-06-04T08:25:18.614 INFO:teuthology.orchestra.run.smithi060.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-06-04T08:25:18.620 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2020-06-04T08:25:18.621 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2020-06-04T08:25:18.883 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'default.rgw.buckets.data' already exists 2020-06-04T08:25:18.893 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2020-06-04T08:25:19.583 INFO:teuthology.orchestra.run.smithi060.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-06-04T08:25:19.596 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2020-06-04T08:25:19.863 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'default.rgw.buckets.index' already exists 2020-06-04T08:25:19.874 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2020-06-04T08:25:20.590 INFO:teuthology.orchestra.run.smithi060.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-06-04T08:25:20.600 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2020-06-04T08:25:20.601 INFO:teuthology.orchestra.run.smithi196:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-06-04T08:25:21.056 INFO:teuthology.orchestra.run.smithi196.stderr:pool 'default.rgw.buckets.data' created 2020-06-04T08:25:21.087 INFO:teuthology.orchestra.run.smithi196:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-06-04T08:25:22.069 INFO:teuthology.orchestra.run.smithi196.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-06-04T08:25:22.085 INFO:teuthology.orchestra.run.smithi196:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-06-04T08:25:23.071 INFO:teuthology.orchestra.run.smithi196.stderr:pool 'default.rgw.buckets.index' created 2020-06-04T08:25:23.093 INFO:teuthology.orchestra.run.smithi196:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-06-04T08:25:24.097 INFO:teuthology.orchestra.run.smithi196.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-06-04T08:25:24.120 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2020-06-04T08:25:24.120 INFO:teuthology.orchestra.run.smithi196:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2020-06-04T08:25:24.375 INFO:teuthology.orchestra.run.smithi196.stderr:pool 'default.rgw.buckets.data' already exists 2020-06-04T08:25:24.385 INFO:teuthology.orchestra.run.smithi196:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2020-06-04T08:25:25.082 INFO:teuthology.orchestra.run.smithi196.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2020-06-04T08:25:25.095 INFO:teuthology.orchestra.run.smithi196:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2020-06-04T08:25:25.374 INFO:teuthology.orchestra.run.smithi196.stderr:pool 'default.rgw.buckets.index' already exists 2020-06-04T08:25:25.393 INFO:teuthology.orchestra.run.smithi196:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2020-06-04T08:25:26.089 INFO:teuthology.orchestra.run.smithi196.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2020-06-04T08:25:26.102 DEBUG:tasks.rgw:Pools created 2020-06-04T08:25:26.103 INFO:tasks.rgw:Configuring compression type = random 2020-06-04T08:25:26.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2020-06-04T08:25:26.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2020-06-04T08:25:26.105 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2020-06-04T08:25:26.144 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:25:26.145 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:25:26.156 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:26.151+0000 7f33ae73e0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:27.631 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.627+0000 7f33ae73e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:27.632 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.627+0000 7f33ae73e0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:27.643 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.639+0000 7f33ae73e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:27.644 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.639+0000 7f33ae73e0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:27.645 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.639+0000 7f33ae73e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:27.645 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.639+0000 7f33ae73e0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:27.645 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.639+0000 7f33ae73e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:27.665 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.663+0000 7f33ae73e0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:27.667 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.663+0000 7f33ae73e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:27.667 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.663+0000 7f33ae73e0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:27.667 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.663+0000 7f33ae73e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:27.668 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.663+0000 7f33ae73e0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:27.668 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.663+0000 7f33ae73e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:27.668 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.663+0000 7f33ae73e0c0 10 cannot find current period zonegroup using local zonegroup 2020-06-04T08:25:27.669 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.663+0000 7f33ae73e0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:27.670 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.663+0000 7f33ae73e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:27.671 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.663+0000 7f33ae73e0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:27.671 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.663+0000 7f33ae73e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:27.672 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.663+0000 7f33ae73e0c0 10 Creating default zonegroup 2020-06-04T08:25:27.673 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.663+0000 7f33ae73e0c0 20 get_system_obj_state: rctx=0x7fff943b1f80 obj=:.pools.avail state=0x562e803614e0 s->prefetch_data=0 2020-06-04T08:25:27.674 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.663+0000 7f33ae73e0c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-06-04T08:25:27.687 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.683+0000 7f33ae73e0c0 10 WARNING: store->list_zones() returned r=-2 2020-06-04T08:25:27.687 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.683+0000 7f33ae73e0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:27.688 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.683+0000 7f33ae73e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:27.692 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.687+0000 7f33ae73e0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:27.694 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.691+0000 7f33ae73e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:27.694 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.691+0000 7f33ae73e0c0 10 could not read realm id: (2) No such file or directory 2020-06-04T08:25:27.694 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.691+0000 7f33ae73e0c0 10 WARNING: failed to set zone as default, r=-22 2020-06-04T08:25:27.695 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.691+0000 7f33ae73e0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:27.695 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.691+0000 7f33ae73e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:27.700 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.695+0000 7f33ae73e0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:27.704 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.695+0000 7f33ae73e0c0 20 rados_obj.operate() r=0 bl.length=839 2020-06-04T08:25:27.705 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.695+0000 7f33ae73e0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:27.705 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.699+0000 7f33ae73e0c0 20 rados_obj.operate() r=0 bl.length=358 2020-06-04T08:25:27.705 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.699+0000 7f33ae73e0c0 20 zonegroup default 2020-06-04T08:25:27.705 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.699+0000 7f33ae73e0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:27.706 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.699+0000 7f33ae73e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:27.706 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.699+0000 7f33ae73e0c0 10 Cannot find current period zone using local zone 2020-06-04T08:25:27.706 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.699+0000 7f33ae73e0c0 10 Using default name default 2020-06-04T08:25:27.706 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.699+0000 7f33ae73e0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:27.707 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.699+0000 7f33ae73e0c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-04T08:25:27.707 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.699+0000 7f33ae73e0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:27.707 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.699+0000 7f33ae73e0c0 20 rados_obj.operate() r=0 bl.length=839 2020-06-04T08:25:27.707 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.699+0000 7f33ae73e0c0 20 zone default found 2020-06-04T08:25:27.707 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:27.699+0000 7f33ae73e0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:29.669 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:29.667+0000 7f33ae73e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:29.670 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:29.667+0000 7f33ae73e0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:29.671 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:29.667+0000 7f33ae73e0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:29.671 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:29.667+0000 7f33ae73e0c0 20 started sync module instance, tier type = 2020-06-04T08:25:29.671 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:29.667+0000 7f33ae73e0c0 20 started zone id=87fd7c76-2d44-477a-8a34-1e9b6372fccb (name=default) with tier type = 2020-06-04T08:25:31.713 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:31.707+0000 7f33ae73e0c0 20 add_watcher() i=0 2020-06-04T08:25:31.713 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:31.707+0000 7f33ae73e0c0 20 add_watcher() i=1 2020-06-04T08:25:31.714 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:31.707+0000 7f33ae73e0c0 20 add_watcher() i=2 2020-06-04T08:25:31.714 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:31.707+0000 7f33ae73e0c0 20 add_watcher() i=3 2020-06-04T08:25:31.714 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:31.707+0000 7f33ae73e0c0 20 add_watcher() i=4 2020-06-04T08:25:31.714 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:31.707+0000 7f33ae73e0c0 20 add_watcher() i=5 2020-06-04T08:25:31.715 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:31.707+0000 7f33ae73e0c0 20 add_watcher() i=6 2020-06-04T08:25:31.715 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:31.711+0000 7f33ae73e0c0 20 add_watcher() i=7 2020-06-04T08:25:31.716 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:31.711+0000 7f33ae73e0c0 2 all 8 watchers are set, enabling cache 2020-06-04T08:25:31.716 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:31.711+0000 7f33ae73e0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-06-04T08:25:31.716 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:31.711+0000 7f33ae73e0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-06-04T08:25:31.717 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:31.711+0000 7f33677e6700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-06-04T08:25:35.693 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.687+0000 7f33ae73e0c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-06-04T08:25:35.810 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.795+0000 7f32d67fc700 20 reqs_thread_entry: start 2020-06-04T08:25:35.810 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.795+0000 7f33ae73e0c0 20 init_complete bucket index max shards: 11 2020-06-04T08:25:35.810 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.795+0000 7f32bffff700 20 reqs_thread_entry: start 2020-06-04T08:25:35.810 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.795+0000 7f32be7fc700 20 INFO: next queue processing will happen at: Thu Jun 4 08:26:05 2020 2020-06-04T08:25:35.811 INFO:teuthology.orchestra.run.smithi060.stderr: 2020-06-04T08:25:35.811 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.795+0000 7f32bdffb700 20 reqs_thread_entry: start 2020-06-04T08:25:35.830 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.827+0000 7f33677e6700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-06-04T08:25:35.833 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.827+0000 7f33ae73e0c0 20 remove_watcher() i=0 2020-06-04T08:25:35.833 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.827+0000 7f33ae73e0c0 2 removed watcher, disabling cache 2020-06-04T08:25:35.835 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.831+0000 7f33ae73e0c0 20 remove_watcher() i=1 2020-06-04T08:25:35.838 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.835+0000 7f33ae73e0c0 20 remove_watcher() i=2 2020-06-04T08:25:35.841 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.835+0000 7f33ae73e0c0 20 remove_watcher() i=3 2020-06-04T08:25:35.844 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.839+0000 7f33ae73e0c0 20 remove_watcher() i=4 2020-06-04T08:25:35.846 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.843+0000 7f33ae73e0c0 20 remove_watcher() i=5 2020-06-04T08:25:35.849 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.847+0000 7f33ae73e0c0 20 remove_watcher() i=6 2020-06-04T08:25:35.859 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.847+0000 7f33ae73e0c0 20 remove_watcher() i=7 2020-06-04T08:25:35.867 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2020-06-04T08:25:35.867 DEBUG:tasks.util.rgw: json result: [] 2020-06-04T08:25:35.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-06-04T08:25:35.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-06-04T08:25:35.870 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-06-04T08:25:35.961 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:25:35.961 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:25:35.977 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.967+0000 7f56395750c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-06-04T08:25:35.978 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.967+0000 7f56395750c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-06-04T08:25:35.979 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.975+0000 7f5569ffb700 20 reqs_thread_entry: start 2020-06-04T08:25:35.980 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.975+0000 7f56395750c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:35.981 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.975+0000 7f56395750c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-04T08:25:35.981 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.975+0000 7f56395750c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:35.982 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.979+0000 7f56395750c0 20 rados_obj.operate() r=0 bl.length=839 2020-06-04T08:25:35.983 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.979+0000 7f56395750c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:35.983 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.979+0000 7f56395750c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:35.984 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.979+0000 7f56395750c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.979+0000 7f56395750c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-04T08:25:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.979+0000 7f56395750c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:35.985 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:35.979+0000 7f56395750c0 20 rados_obj.operate() r=0 bl.length=358 2020-06-04T08:25:35.995 INFO:teuthology.orchestra.run.smithi060.stdout:{"id":"87fd7c76-2d44-477a-8a34-1e9b6372fccb","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2020-06-04T08:25:35.996 DEBUG:tasks.util.rgw: json result: {'id': '87fd7c76-2d44-477a-8a34-1e9b6372fccb', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-06-04T08:25:35.997 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2020-06-04T08:25:35.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2020-06-04T08:25:35.998 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2020-06-04T08:25:36.076 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:25:36.076 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:25:36.088 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.083+0000 7f9e8a3f40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.089 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.087+0000 7f9e8a3f40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:36.089 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.087+0000 7f9e8a3f40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.090 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.087+0000 7f9e8a3f40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:36.090 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.087+0000 7f9e8a3f40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.091 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.087+0000 7f9e8a3f40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:36.091 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.087+0000 7f9e8a3f40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.091 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.087+0000 7f9e8a3f40c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-04T08:25:36.119 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.115+0000 7f9e8a3f40c0 20 RGWRados::pool_iterate: got zone_info.87fd7c76-2d44-477a-8a34-1e9b6372fccb 2020-06-04T08:25:36.119 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.115+0000 7f9e8a3f40c0 20 RGWRados::pool_iterate: got zonegroup_info.304c8299-f3af-492a-89aa-f54619f993d7 2020-06-04T08:25:36.120 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.115+0000 7f9e8a3f40c0 20 RGWRados::pool_iterate: got zone_names.default 2020-06-04T08:25:36.120 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.115+0000 7f9e8a3f40c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-06-04T08:25:36.120 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.115+0000 7f9e8a3f40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.121 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.115+0000 7f9e8a3f40c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-04T08:25:36.121 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.115+0000 7f9e8a3f40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.123 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.115+0000 7f9e8a3f40c0 20 rados_obj.operate() r=0 bl.length=855 2020-06-04T08:25:36.123 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.115+0000 7f9e8a3f40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.125 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.119+0000 7f9e8a3f40c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-04T08:25:36.126 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.119+0000 7f9e8a3f40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.126 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.119+0000 7f9e8a3f40c0 20 rados_obj.operate() r=0 bl.length=358 2020-06-04T08:25:36.127 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.119+0000 7f9e8a3f40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.127 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.119+0000 7f9e8a3f40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:36.127 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.119+0000 7f9e8a3f40c0 10 cannot find current period zonegroup using local zonegroup 2020-06-04T08:25:36.127 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.119+0000 7f9e8a3f40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.128 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.119+0000 7f9e8a3f40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:36.128 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.119+0000 7f9e8a3f40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.128 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.123+0000 7f9e8a3f40c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-04T08:25:36.129 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.123+0000 7f9e8a3f40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.132 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.123+0000 7f9e8a3f40c0 20 rados_obj.operate() r=0 bl.length=358 2020-06-04T08:25:36.132 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.123+0000 7f9e8a3f40c0 20 zonegroup default 2020-06-04T08:25:36.132 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.123+0000 7f9e8a3f40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.133 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.123+0000 7f9e8a3f40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:36.133 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.123+0000 7f9e8a3f40c0 10 Cannot find current period zone using local zone 2020-06-04T08:25:36.133 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.123+0000 7f9e8a3f40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.133 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.123+0000 7f9e8a3f40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:36.133 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.123+0000 7f9e8a3f40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.134 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.123+0000 7f9e8a3f40c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-04T08:25:36.134 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.123+0000 7f9e8a3f40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.134 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.127+0000 7f9e8a3f40c0 20 rados_obj.operate() r=0 bl.length=855 2020-06-04T08:25:36.134 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.127+0000 7f9e8a3f40c0 20 zone default found 2020-06-04T08:25:36.134 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.127+0000 7f9e8a3f40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.134 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.127+0000 7f9e8a3f40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:36.135 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.127+0000 7f9e8a3f40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.135 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.127+0000 7f9e8a3f40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:36.135 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.127+0000 7f9e8a3f40c0 20 started sync module instance, tier type = 2020-06-04T08:25:36.135 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.127+0000 7f9e8a3f40c0 20 started zone id=87fd7c76-2d44-477a-8a34-1e9b6372fccb (name=default) with tier type = 2020-06-04T08:25:36.155 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.151+0000 7f9e8a3f40c0 20 add_watcher() i=0 2020-06-04T08:25:36.155 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.151+0000 7f9e8a3f40c0 20 add_watcher() i=1 2020-06-04T08:25:36.155 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.151+0000 7f9e8a3f40c0 20 add_watcher() i=2 2020-06-04T08:25:36.156 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.151+0000 7f9e8a3f40c0 20 add_watcher() i=3 2020-06-04T08:25:36.156 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.151+0000 7f9e8a3f40c0 20 add_watcher() i=4 2020-06-04T08:25:36.156 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.151+0000 7f9e8a3f40c0 20 add_watcher() i=5 2020-06-04T08:25:36.158 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.151+0000 7f9e8a3f40c0 20 add_watcher() i=6 2020-06-04T08:25:36.158 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.151+0000 7f9e8a3f40c0 20 add_watcher() i=7 2020-06-04T08:25:36.159 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.151+0000 7f9e8a3f40c0 2 all 8 watchers are set, enabling cache 2020-06-04T08:25:36.159 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.151+0000 7f9e8a3f40c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-06-04T08:25:36.159 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.151+0000 7f9e8a3f40c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-06-04T08:25:36.160 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.151+0000 7f9e437e6700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-06-04T08:25:36.251 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.247+0000 7f9dba7fc700 20 reqs_thread_entry: start 2020-06-04T08:25:36.251 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.247+0000 7f9e8a3f40c0 20 init_complete bucket index max shards: 11 2020-06-04T08:25:36.251 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.247+0000 7f9da7fff700 20 reqs_thread_entry: start 2020-06-04T08:25:36.252 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.247+0000 7f9da67fc700 20 INFO: next queue processing will happen at: Thu Jun 4 08:26:06 2020 2020-06-04T08:25:36.252 INFO:teuthology.orchestra.run.smithi060.stderr: 2020-06-04T08:25:36.252 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.247+0000 7f9da5ffb700 20 reqs_thread_entry: start 2020-06-04T08:25:36.281 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.275+0000 7f9e437e6700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-06-04T08:25:36.289 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.275+0000 7f9e8a3f40c0 20 remove_watcher() i=0 2020-06-04T08:25:36.289 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.275+0000 7f9e8a3f40c0 2 removed watcher, disabling cache 2020-06-04T08:25:36.290 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.279+0000 7f9e8a3f40c0 20 remove_watcher() i=1 2020-06-04T08:25:36.290 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.283+0000 7f9e8a3f40c0 20 remove_watcher() i=2 2020-06-04T08:25:36.291 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.283+0000 7f9e8a3f40c0 20 remove_watcher() i=3 2020-06-04T08:25:36.293 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.287+0000 7f9e8a3f40c0 20 remove_watcher() i=4 2020-06-04T08:25:36.306 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.291+0000 7f9e8a3f40c0 20 remove_watcher() i=5 2020-06-04T08:25:36.306 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.291+0000 7f9e8a3f40c0 20 remove_watcher() i=6 2020-06-04T08:25:36.307 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.295+0000 7f9e8a3f40c0 20 remove_watcher() i=7 2020-06-04T08:25:36.307 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2020-06-04T08:25:36.308 DEBUG:tasks.util.rgw: json result: [] 2020-06-04T08:25:36.308 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-06-04T08:25:36.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-06-04T08:25:36.309 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-06-04T08:25:36.394 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:25:36.394 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:25:36.410 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.403+0000 7f9b86ad70c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-06-04T08:25:36.410 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.403+0000 7f9b86ad70c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-06-04T08:25:36.419 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.411+0000 7f9aaffff700 20 reqs_thread_entry: start 2020-06-04T08:25:36.419 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.411+0000 7f9b86ad70c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.419 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.411+0000 7f9b86ad70c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-04T08:25:36.419 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.411+0000 7f9b86ad70c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.420 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.411+0000 7f9b86ad70c0 20 rados_obj.operate() r=0 bl.length=855 2020-06-04T08:25:36.420 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.411+0000 7f9b86ad70c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.420 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.415+0000 7f9b86ad70c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:36.420 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.415+0000 7f9b86ad70c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.420 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.415+0000 7f9b86ad70c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-04T08:25:36.421 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.415+0000 7f9b86ad70c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:36.421 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:36.415+0000 7f9b86ad70c0 20 rados_obj.operate() r=0 bl.length=358 2020-06-04T08:25:36.429 INFO:teuthology.orchestra.run.smithi060.stdout:{"id":"87fd7c76-2d44-477a-8a34-1e9b6372fccb","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2020-06-04T08:25:36.430 DEBUG:tasks.util.rgw: json result: {'id': '87fd7c76-2d44-477a-8a34-1e9b6372fccb', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-06-04T08:25:36.430 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2020-06-04T08:25:36.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2020-06-04T08:25:36.431 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2020-06-04T08:25:36.466 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:25:36.467 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:25:36.479 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:36.479+0000 7fb1663ea0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:37.934 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.935+0000 7fb1663ea0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:37.935 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.935+0000 7fb1663ea0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:37.946 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.939+0000 7fb1663ea0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:37.947 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.939+0000 7fb1663ea0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:37.947 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.939+0000 7fb1663ea0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:37.947 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.939+0000 7fb1663ea0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:37.947 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.939+0000 7fb1663ea0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:37.968 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.959+0000 7fb1663ea0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:37.969 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.959+0000 7fb1663ea0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:37.969 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.959+0000 7fb1663ea0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:37.980 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.959+0000 7fb1663ea0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:37.981 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.959+0000 7fb1663ea0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:37.981 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.963+0000 7fb1663ea0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:37.994 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.963+0000 7fb1663ea0c0 10 cannot find current period zonegroup using local zonegroup 2020-06-04T08:25:37.995 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.963+0000 7fb1663ea0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:37.995 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.963+0000 7fb1663ea0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:38.002 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.963+0000 7fb1663ea0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:38.002 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.963+0000 7fb1663ea0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:38.004 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.963+0000 7fb1663ea0c0 10 Creating default zonegroup 2020-06-04T08:25:38.004 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.963+0000 7fb1663ea0c0 20 get_system_obj_state: rctx=0x7ffd02d13790 obj=:.pools.avail state=0x55c69ead14e0 s->prefetch_data=0 2020-06-04T08:25:38.004 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.963+0000 7fb1663ea0c0 10 couldn't find old data placement pools config, setting up new ones for the zone 2020-06-04T08:25:38.007 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.979+0000 7fb1663ea0c0 10 WARNING: store->list_zones() returned r=-2 2020-06-04T08:25:38.008 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.979+0000 7fb1663ea0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:38.008 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.979+0000 7fb1663ea0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:38.009 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.983+0000 7fb1663ea0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:38.009 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.987+0000 7fb1663ea0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:38.009 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.987+0000 7fb1663ea0c0 10 could not read realm id: (2) No such file or directory 2020-06-04T08:25:38.025 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.987+0000 7fb1663ea0c0 10 WARNING: failed to set zone as default, r=-22 2020-06-04T08:25:38.026 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.987+0000 7fb1663ea0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:38.026 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.987+0000 7fb1663ea0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:38.027 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.991+0000 7fb1663ea0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:38.027 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.991+0000 7fb1663ea0c0 20 rados_obj.operate() r=0 bl.length=839 2020-06-04T08:25:38.027 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.991+0000 7fb1663ea0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:38.027 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.995+0000 7fb1663ea0c0 20 rados_obj.operate() r=0 bl.length=358 2020-06-04T08:25:38.028 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.995+0000 7fb1663ea0c0 20 zonegroup default 2020-06-04T08:25:38.028 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.995+0000 7fb1663ea0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:38.028 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.995+0000 7fb1663ea0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:38.028 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.995+0000 7fb1663ea0c0 10 Cannot find current period zone using local zone 2020-06-04T08:25:38.029 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.995+0000 7fb1663ea0c0 10 Using default name default 2020-06-04T08:25:38.029 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.995+0000 7fb1663ea0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:38.029 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.995+0000 7fb1663ea0c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-04T08:25:38.029 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.995+0000 7fb1663ea0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:38.030 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.995+0000 7fb1663ea0c0 20 rados_obj.operate() r=0 bl.length=839 2020-06-04T08:25:38.030 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.995+0000 7fb1663ea0c0 20 zone default found 2020-06-04T08:25:38.030 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:37.995+0000 7fb1663ea0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:39.958 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:39.959+0000 7fb1663ea0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:39.959 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:39.959+0000 7fb1663ea0c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:39.966 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:39.967+0000 7fb1663ea0c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:39.966 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:39.967+0000 7fb1663ea0c0 20 started sync module instance, tier type = 2020-06-04T08:25:39.967 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:39.967+0000 7fb1663ea0c0 20 started zone id=e2c8f8c0-91bb-4e72-ac03-5b74d6df4799 (name=default) with tier type = 2020-06-04T08:25:41.999 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:41.995+0000 7fb1663ea0c0 20 add_watcher() i=0 2020-06-04T08:25:42.000 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:41.995+0000 7fb1663ea0c0 20 add_watcher() i=1 2020-06-04T08:25:42.000 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:41.995+0000 7fb1663ea0c0 20 add_watcher() i=2 2020-06-04T08:25:42.000 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:41.995+0000 7fb1663ea0c0 20 add_watcher() i=3 2020-06-04T08:25:42.000 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:41.995+0000 7fb1663ea0c0 20 add_watcher() i=4 2020-06-04T08:25:42.001 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:41.995+0000 7fb1663ea0c0 20 add_watcher() i=5 2020-06-04T08:25:42.001 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:41.995+0000 7fb1663ea0c0 20 add_watcher() i=6 2020-06-04T08:25:42.001 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:41.999+0000 7fb1663ea0c0 20 add_watcher() i=7 2020-06-04T08:25:42.001 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:41.999+0000 7fb1663ea0c0 2 all 8 watchers are set, enabling cache 2020-06-04T08:25:42.002 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:41.999+0000 7fb1663ea0c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-06-04T08:25:42.002 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:41.999+0000 7fb1663ea0c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-06-04T08:25:42.002 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:41.999+0000 7fb11f7e6700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-06-04T08:25:45.970 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:45.966+0000 7fb1663ea0c0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2020-06-04T08:25:46.149 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.146+0000 7fb08e7fc700 20 reqs_thread_entry: start 2020-06-04T08:25:46.149 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.146+0000 7fb1663ea0c0 20 init_complete bucket index max shards: 11 2020-06-04T08:25:46.150 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.146+0000 7fb077fff700 20 reqs_thread_entry: start 2020-06-04T08:25:46.150 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.146+0000 7fb0767fc700 20 INFO: next queue processing will happen at: Thu Jun 4 08:26:16 2020 2020-06-04T08:25:46.150 INFO:teuthology.orchestra.run.smithi196.stderr: 2020-06-04T08:25:46.150 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.146+0000 7fb075ffb700 20 reqs_thread_entry: start 2020-06-04T08:25:46.179 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.170+0000 7fb11f7e6700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-06-04T08:25:46.179 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.174+0000 7fb1663ea0c0 20 remove_watcher() i=0 2020-06-04T08:25:46.179 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.174+0000 7fb1663ea0c0 2 removed watcher, disabling cache 2020-06-04T08:25:46.180 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.178+0000 7fb1663ea0c0 20 remove_watcher() i=1 2020-06-04T08:25:46.186 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.178+0000 7fb1663ea0c0 20 remove_watcher() i=2 2020-06-04T08:25:46.187 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.182+0000 7fb1663ea0c0 20 remove_watcher() i=3 2020-06-04T08:25:46.187 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.182+0000 7fb1663ea0c0 20 remove_watcher() i=4 2020-06-04T08:25:46.194 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.186+0000 7fb1663ea0c0 20 remove_watcher() i=5 2020-06-04T08:25:46.194 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.190+0000 7fb1663ea0c0 20 remove_watcher() i=6 2020-06-04T08:25:46.195 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.190+0000 7fb1663ea0c0 20 remove_watcher() i=7 2020-06-04T08:25:46.215 INFO:teuthology.orchestra.run.smithi196.stdout:[] 2020-06-04T08:25:46.215 DEBUG:tasks.util.rgw: json result: [] 2020-06-04T08:25:46.216 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-06-04T08:25:46.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-06-04T08:25:46.218 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-06-04T08:25:46.301 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:25:46.302 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:25:46.309 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.306+0000 7fc75a7c80c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-06-04T08:25:46.310 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.306+0000 7fc75a7c80c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-06-04T08:25:46.314 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.310+0000 7fc75a7c80c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.315 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.310+0000 7fc68b7fe700 20 reqs_thread_entry: start 2020-06-04T08:25:46.316 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.314+0000 7fc75a7c80c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-04T08:25:46.316 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.314+0000 7fc75a7c80c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.317 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.314+0000 7fc75a7c80c0 20 rados_obj.operate() r=0 bl.length=839 2020-06-04T08:25:46.318 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.314+0000 7fc75a7c80c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.318 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.314+0000 7fc75a7c80c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:46.318 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.314+0000 7fc75a7c80c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.319 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.314+0000 7fc75a7c80c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-04T08:25:46.319 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.314+0000 7fc75a7c80c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.320 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.318+0000 7fc75a7c80c0 20 rados_obj.operate() r=0 bl.length=358 2020-06-04T08:25:46.329 INFO:teuthology.orchestra.run.smithi196.stdout:{"id":"e2c8f8c0-91bb-4e72-ac03-5b74d6df4799","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2020-06-04T08:25:46.330 DEBUG:tasks.util.rgw: json result: {'id': 'e2c8f8c0-91bb-4e72-ac03-5b74d6df4799', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-06-04T08:25:46.330 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2020-06-04T08:25:46.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2020-06-04T08:25:46.331 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2020-06-04T08:25:46.406 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:25:46.407 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:25:46.412 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.406+0000 7f9e9c8380c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.412 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.406+0000 7f9e9c8380c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:46.413 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.406+0000 7f9e9c8380c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.413 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.410+0000 7f9e9c8380c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:46.413 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.410+0000 7f9e9c8380c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.413 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.410+0000 7f9e9c8380c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:46.413 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.410+0000 7f9e9c8380c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.414 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.410+0000 7f9e9c8380c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-04T08:25:46.427 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.422+0000 7f9e9c8380c0 20 RGWRados::pool_iterate: got zone_info.e2c8f8c0-91bb-4e72-ac03-5b74d6df4799 2020-06-04T08:25:46.427 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.422+0000 7f9e9c8380c0 20 RGWRados::pool_iterate: got zonegroup_info.4a777b23-56cf-46f7-8203-c4e921f6be2f 2020-06-04T08:25:46.427 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.422+0000 7f9e9c8380c0 20 RGWRados::pool_iterate: got zone_names.default 2020-06-04T08:25:46.428 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.422+0000 7f9e9c8380c0 20 RGWRados::pool_iterate: got zonegroups_names.default 2020-06-04T08:25:46.428 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.422+0000 7f9e9c8380c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.428 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.426+0000 7f9e9c8380c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-04T08:25:46.429 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.426+0000 7f9e9c8380c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.430 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.426+0000 7f9e9c8380c0 20 rados_obj.operate() r=0 bl.length=855 2020-06-04T08:25:46.430 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.426+0000 7f9e9c8380c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.431 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.426+0000 7f9e9c8380c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-04T08:25:46.431 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.426+0000 7f9e9c8380c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.431 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.426+0000 7f9e9c8380c0 20 rados_obj.operate() r=0 bl.length=358 2020-06-04T08:25:46.431 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.426+0000 7f9e9c8380c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.432 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.426+0000 7f9e9c8380c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:46.432 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.426+0000 7f9e9c8380c0 10 cannot find current period zonegroup using local zonegroup 2020-06-04T08:25:46.432 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.426+0000 7f9e9c8380c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.432 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.426+0000 7f9e9c8380c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:46.432 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.426+0000 7f9e9c8380c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.434 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.430+0000 7f9e9c8380c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-04T08:25:46.434 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.430+0000 7f9e9c8380c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.435 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.430+0000 7f9e9c8380c0 20 rados_obj.operate() r=0 bl.length=358 2020-06-04T08:25:46.435 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.430+0000 7f9e9c8380c0 20 zonegroup default 2020-06-04T08:25:46.435 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.430+0000 7f9e9c8380c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.435 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.430+0000 7f9e9c8380c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:46.435 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.430+0000 7f9e9c8380c0 10 Cannot find current period zone using local zone 2020-06-04T08:25:46.436 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.430+0000 7f9e9c8380c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.436 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.430+0000 7f9e9c8380c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:46.436 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.430+0000 7f9e9c8380c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.436 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.430+0000 7f9e9c8380c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-04T08:25:46.436 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.430+0000 7f9e9c8380c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.436 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.430+0000 7f9e9c8380c0 20 rados_obj.operate() r=0 bl.length=855 2020-06-04T08:25:46.437 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.430+0000 7f9e9c8380c0 20 zone default found 2020-06-04T08:25:46.437 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.430+0000 7f9e9c8380c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.437 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.430+0000 7f9e9c8380c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:46.437 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.430+0000 7f9e9c8380c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.437 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.430+0000 7f9e9c8380c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:46.438 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.430+0000 7f9e9c8380c0 20 started sync module instance, tier type = 2020-06-04T08:25:46.438 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.430+0000 7f9e9c8380c0 20 started zone id=e2c8f8c0-91bb-4e72-ac03-5b74d6df4799 (name=default) with tier type = 2020-06-04T08:25:46.451 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.450+0000 7f9e9c8380c0 20 add_watcher() i=0 2020-06-04T08:25:46.451 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.450+0000 7f9e9c8380c0 20 add_watcher() i=1 2020-06-04T08:25:46.452 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.450+0000 7f9e9c8380c0 20 add_watcher() i=2 2020-06-04T08:25:46.452 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.450+0000 7f9e9c8380c0 20 add_watcher() i=3 2020-06-04T08:25:46.452 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.450+0000 7f9e9c8380c0 20 add_watcher() i=4 2020-06-04T08:25:46.452 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.450+0000 7f9e9c8380c0 20 add_watcher() i=5 2020-06-04T08:25:46.453 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.450+0000 7f9e9c8380c0 20 add_watcher() i=6 2020-06-04T08:25:46.453 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.450+0000 7f9e9c8380c0 20 add_watcher() i=7 2020-06-04T08:25:46.453 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.450+0000 7f9e9c8380c0 2 all 8 watchers are set, enabling cache 2020-06-04T08:25:46.454 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.450+0000 7f9e9c8380c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-06-04T08:25:46.454 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.450+0000 7f9e9c8380c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-06-04T08:25:46.454 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.450+0000 7f9e557ea700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-06-04T08:25:46.531 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.530+0000 7f9dbffff700 20 reqs_thread_entry: start 2020-06-04T08:25:46.532 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.530+0000 7f9dbdffb700 20 reqs_thread_entry: start 2020-06-04T08:25:46.532 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.530+0000 7f9e9c8380c0 20 init_complete bucket index max shards: 11 2020-06-04T08:25:46.532 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.530+0000 7f9daffff700 20 INFO: next queue processing will happen at: Thu Jun 4 08:26:16 2020 2020-06-04T08:25:46.533 INFO:teuthology.orchestra.run.smithi196.stderr: 2020-06-04T08:25:46.533 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.530+0000 7f9daf7fe700 20 reqs_thread_entry: start 2020-06-04T08:25:46.559 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.558+0000 7f9e557ea700 2 RGWDataChangesLog::ChangesRenewThread: start 2020-06-04T08:25:46.562 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.558+0000 7f9e9c8380c0 20 remove_watcher() i=0 2020-06-04T08:25:46.562 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.558+0000 7f9e9c8380c0 2 removed watcher, disabling cache 2020-06-04T08:25:46.565 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.562+0000 7f9e9c8380c0 20 remove_watcher() i=1 2020-06-04T08:25:46.568 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.566+0000 7f9e9c8380c0 20 remove_watcher() i=2 2020-06-04T08:25:46.571 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.570+0000 7f9e9c8380c0 20 remove_watcher() i=3 2020-06-04T08:25:46.574 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.570+0000 7f9e9c8380c0 20 remove_watcher() i=4 2020-06-04T08:25:46.576 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.574+0000 7f9e9c8380c0 20 remove_watcher() i=5 2020-06-04T08:25:46.579 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.578+0000 7f9e9c8380c0 20 remove_watcher() i=6 2020-06-04T08:25:46.581 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.578+0000 7f9e9c8380c0 20 remove_watcher() i=7 2020-06-04T08:25:46.596 INFO:teuthology.orchestra.run.smithi196.stdout:[] 2020-06-04T08:25:46.596 DEBUG:tasks.util.rgw: json result: [] 2020-06-04T08:25:46.597 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-06-04T08:25:46.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2020-06-04T08:25:46.597 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2020-06-04T08:25:46.677 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:25:46.678 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:25:46.702 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.682+0000 7fab1ddf40c0 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2020-06-04T08:25:46.702 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.682+0000 7fab1ddf40c0 20 check_secure_mon_conn(): mode 1 is insecure 2020-06-04T08:25:46.702 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.690+0000 7fab1ddf40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.714 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.690+0000 7faa4effd700 20 reqs_thread_entry: start 2020-06-04T08:25:46.714 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.694+0000 7fab1ddf40c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-04T08:25:46.714 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.694+0000 7fab1ddf40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.715 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.694+0000 7fab1ddf40c0 20 rados_obj.operate() r=0 bl.length=855 2020-06-04T08:25:46.715 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.694+0000 7fab1ddf40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.716 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.694+0000 7fab1ddf40c0 20 rados_obj.operate() r=-2 bl.length=0 2020-06-04T08:25:46.716 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.694+0000 7fab1ddf40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.716 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.694+0000 7fab1ddf40c0 20 rados_obj.operate() r=0 bl.length=46 2020-06-04T08:25:46.717 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.694+0000 7fab1ddf40c0 20 rados->read ofs=0 len=0 2020-06-04T08:25:46.717 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:25:46.698+0000 7fab1ddf40c0 20 rados_obj.operate() r=0 bl.length=358 2020-06-04T08:25:46.719 INFO:teuthology.orchestra.run.smithi196.stdout:{"id":"e2c8f8c0-91bb-4e72-ac03-5b74d6df4799","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2020-06-04T08:25:46.719 DEBUG:tasks.util.rgw: json result: {'id': 'e2c8f8c0-91bb-4e72-ac03-5b74d6df4799', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2020-06-04T08:25:46.720 INFO:tasks.rgw:Starting rgw... 2020-06-04T08:25:46.721 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2020-06-04T08:25:46.721 INFO:tasks.rgw:Using beast as radosgw frontend 2020-06-04T08:25:46.722 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-06-04T08:25:46.723 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-06-04T08:25:46.727 INFO:tasks.rgw.c1.client.0:Started 2020-06-04T08:25:46.727 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2020-06-04T08:25:46.737 INFO:tasks.rgw:Using beast as radosgw frontend 2020-06-04T08:25:46.737 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-06-04T08:25:46.737 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-06-04T08:25:46.761 INFO:tasks.rgw.c1.client.1:Started 2020-06-04T08:25:46.762 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2020-06-04T08:25:46.762 INFO:tasks.rgw:Using beast as radosgw frontend 2020-06-04T08:25:46.762 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-06-04T08:25:46.763 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-06-04T08:25:46.765 INFO:tasks.rgw.c2.client.0:Started 2020-06-04T08:25:46.766 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2020-06-04T08:25:46.766 INFO:tasks.rgw:Using beast as radosgw frontend 2020-06-04T08:25:46.767 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-06-04T08:25:46.767 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-06-04T08:25:46.770 INFO:tasks.rgw.c2.client.1:Started 2020-06-04T08:25:46.770 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi060.front.sepia.ceph.com:8000/ 2020-06-04T08:25:46.771 INFO:teuthology.orchestra.run.smithi060:> curl http://smithi060.front.sepia.ceph.com:8000/ 2020-06-04T08:25:46.787 INFO:teuthology.orchestra.run.smithi060.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-04T08:25:46.788 INFO:teuthology.orchestra.run.smithi060.stderr: Dload Upload Total Spent Left Speed 2020-06-04T08:25:46.791 INFO:teuthology.orchestra.run.smithi060.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi060.front.sepia.ceph.com port 8000: Connection refused 2020-06-04T08:25:46.793 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-06-04T08:25:47.795 INFO:teuthology.orchestra.run.smithi060:> curl http://smithi060.front.sepia.ceph.com:8000/ 2020-06-04T08:25:47.819 INFO:teuthology.orchestra.run.smithi060.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-04T08:25:47.820 INFO:teuthology.orchestra.run.smithi060.stderr: Dload Upload Total Spent Left Speed 2020-06-04T08:25:47.824 INFO:teuthology.orchestra.run.smithi060.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-06-04T08:25:47.825 INFO:teuthology.orchestra.run.smithi060.stdout:anonymous 2020-06-04T08:25:47.831 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi060.front.sepia.ceph.com:8001/ 2020-06-04T08:25:47.832 INFO:teuthology.orchestra.run.smithi060:> curl http://smithi060.front.sepia.ceph.com:8001/ 2020-06-04T08:25:47.886 INFO:teuthology.orchestra.run.smithi060.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-04T08:25:47.886 INFO:teuthology.orchestra.run.smithi060.stderr: Dload Upload Total Spent Left Speed 2020-06-04T08:25:47.892 INFO:teuthology.orchestra.run.smithi060.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-06-04T08:25:47.892 INFO:teuthology.orchestra.run.smithi060.stdout:anonymous 2020-06-04T08:25:47.893 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi196.front.sepia.ceph.com:8000/ 2020-06-04T08:25:47.894 INFO:teuthology.orchestra.run.smithi196:> curl http://smithi196.front.sepia.ceph.com:8000/ 2020-06-04T08:25:47.913 INFO:teuthology.orchestra.run.smithi196.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-04T08:25:47.913 INFO:teuthology.orchestra.run.smithi196.stderr: Dload Upload Total Spent Left Speed 2020-06-04T08:25:47.919 INFO:teuthology.orchestra.run.smithi196.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 19454 0 --:--:-- --:--:-- --:--:-- 42800 2020-06-04T08:25:47.919 INFO:teuthology.orchestra.run.smithi196.stdout:anonymous 2020-06-04T08:25:47.919 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi196.front.sepia.ceph.com:8001/ 2020-06-04T08:25:47.920 INFO:teuthology.orchestra.run.smithi196:> curl http://smithi196.front.sepia.ceph.com:8001/ 2020-06-04T08:25:47.979 INFO:teuthology.orchestra.run.smithi196.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-04T08:25:47.979 INFO:teuthology.orchestra.run.smithi196.stderr: Dload Upload Total Spent Left Speed 2020-06-04T08:25:47.983 INFO:teuthology.orchestra.run.smithi196.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-06-04T08:25:47.984 INFO:teuthology.orchestra.run.smithi196.stdout:anonymous 2020-06-04T08:25:47.985 INFO:teuthology.run_tasks:Running task rgw-multisite... 2020-06-04T08:25:48.248 INFO:tasks.rgw_multisite:creating realm.. 2020-06-04T08:25:48.249 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:25:48.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:25:48.250 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:25:48.291 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:25:48.292 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:25:48.350 INFO:teuthology.orchestra.run.smithi060.stdout:{"id":"9c559937-87e1-4867-81f3-e687b7a20f8b","name":"test-realm","current_period":"e080917d-3ba9-4cac-ad11-9299b0b52e3e","epoch":1} 2020-06-04T08:25:48.351 INFO:tasks.rgw_multisite:creating master zonegroup.. 2020-06-04T08:25:48.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:25:48.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:25:48.352 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:25:48.434 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:25:48.434 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:25:48.454 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:25:48.451+0000 7f50cc8210c0 -1 old format 2020-06-04T08:25:48.473 INFO:teuthology.orchestra.run.smithi060.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","sync_policy":{"groups":[]}} 2020-06-04T08:25:48.474 INFO:tasks.rgw_multisite:creating master zone.. 2020-06-04T08:25:48.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'M2KEYTP439P9KXXD', '--secret', 'Ad56bMSsbqPkjfMTVush7dBgYdLf27ul', '--endpoints', 'http://smithi060.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:25:48.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'M2KEYTP439P9KXXD', '--secret', 'Ad56bMSsbqPkjfMTVush7dBgYdLf27ul', '--endpoints', 'http://smithi060.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:25:48.475 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key M2KEYTP439P9KXXD --secret Ad56bMSsbqPkjfMTVush7dBgYdLf27ul --endpoints http://smithi060.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:25:48.567 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:25:48.568 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:25:48.629 INFO:teuthology.orchestra.run.smithi060.stdout:{"id":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"M2KEYTP439P9KXXD","secret_key":"Ad56bMSsbqPkjfMTVush7dBgYdLf27ul"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","notif_pool":"test-zone1.rgw.log:notif"} 2020-06-04T08:25:48.633 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2020-06-04T08:25:49.733 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'test-zone1.rgw.buckets.data' created 2020-06-04T08:25:49.748 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2020-06-04T08:25:50.742 INFO:teuthology.orchestra.run.smithi060.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2020-06-04T08:25:50.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:25:50.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:25:50.771 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:25:50.814 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:25:50.814 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:25:50.842 INFO:teuthology.orchestra.run.smithi060.stdout:{"id":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"M2KEYTP439P9KXXD","secret_key":"Ad56bMSsbqPkjfMTVush7dBgYdLf27ul"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"test-zone1.rgw.log:notif"} 2020-06-04T08:25:50.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:25:50.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:25:50.844 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:25:50.921 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:25:50.921 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:25:58.996 INFO:teuthology.orchestra.run.smithi060.stdout:{"id":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","epoch":1,"predecessor_uuid":"e080917d-3ba9-4cac-ad11-9299b0b52e3e","sync_status":[],"period_map":{"id":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","zones":[{"id":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","name":"test-zone1","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","val":3611466229}]},"master_zonegroup":"test-zonegroup","master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","realm_name":"test-realm","realm_epoch":2} 2020-06-04T08:25:58.998 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2020-06-04T08:25:58.998 INFO:teuthology.orchestra.run:waiting for 300 2020-06-04T08:25:59.039 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-06-04T08:25:59.035+0000 7fe889731700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 15794) UID: 0 2020-06-04T08:25:59.039 INFO:tasks.rgw.c1.client.0.smithi060.stdout:2020-06-04T08:25:59.035+0000 7fe8a0dc20c0 -1 shutting down 2020-06-04T08:26:04.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2020-06-04T08:26:05.100 INFO:tasks.rgw.c1.client.0:Stopped 2020-06-04T08:26:05.100 INFO:tasks.rgw.c1.client.0:Restarting daemon 2020-06-04T08:26:05.101 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2020-06-04T08:26:05.104 INFO:tasks.rgw.c1.client.0:Started 2020-06-04T08:26:05.104 INFO:teuthology.orchestra.run.smithi060:> curl http://smithi060.front.sepia.ceph.com:8000 2020-06-04T08:26:05.122 INFO:teuthology.orchestra.run.smithi060.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-04T08:26:05.123 INFO:teuthology.orchestra.run.smithi060.stderr: Dload Upload Total Spent Left Speed 2020-06-04T08:26:05.127 INFO:teuthology.orchestra.run.smithi060.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi060.front.sepia.ceph.com port 8000: Connection refused 2020-06-04T08:26:05.128 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-06-04T08:26:06.130 INFO:teuthology.orchestra.run.smithi060:> curl http://smithi060.front.sepia.ceph.com:8000 2020-06-04T08:26:06.152 INFO:teuthology.orchestra.run.smithi060.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-04T08:26:06.152 INFO:teuthology.orchestra.run.smithi060.stderr: Dload Upload Total Spent Left Speed 2020-06-04T08:26:06.153 INFO:teuthology.orchestra.run.smithi060.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2020-06-04T08:26:06.153 INFO:teuthology.orchestra.run.smithi060.stdout:anonymous 2020-06-04T08:26:06.153 INFO:tasks.rgw_multisite:creating admin user.. 2020-06-04T08:26:06.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'M2KEYTP439P9KXXD', '--secret', 'Ad56bMSsbqPkjfMTVush7dBgYdLf27ul', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:06.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'M2KEYTP439P9KXXD', '--secret', 'Ad56bMSsbqPkjfMTVush7dBgYdLf27ul', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:06.155 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key M2KEYTP439P9KXXD --secret Ad56bMSsbqPkjfMTVush7dBgYdLf27ul --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:26:06.245 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:26:06.245 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:26:06.426 INFO:teuthology.orchestra.run.smithi060.stdout:{"user_id":"realm-admin","display_name":"Realm Admin","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"realm-admin","access_key":"M2KEYTP439P9KXXD","secret_key":"Ad56bMSsbqPkjfMTVush7dBgYdLf27ul"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","system":"true","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2020-06-04T08:26:06.455 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-06-04T08:26:06.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi060.front.sepia.ceph.com:8000', '--access-key', 'M2KEYTP439P9KXXD', '--secret', 'Ad56bMSsbqPkjfMTVush7dBgYdLf27ul', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:06.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi060.front.sepia.ceph.com:8000', '--access-key', 'M2KEYTP439P9KXXD', '--secret', 'Ad56bMSsbqPkjfMTVush7dBgYdLf27ul', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:06.457 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi060.front.sepia.ceph.com:8000 --access-key M2KEYTP439P9KXXD --secret Ad56bMSsbqPkjfMTVush7dBgYdLf27ul --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:26:06.512 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:26:06.513 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:26:06.622 INFO:teuthology.orchestra.run.smithi196.stdout:{"id":"9c559937-87e1-4867-81f3-e687b7a20f8b","name":"test-realm","current_period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","epoch":2} 2020-06-04T08:26:06.623 INFO:tasks.rgw_multisite:creating zone.. 2020-06-04T08:26:06.624 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'M2KEYTP439P9KXXD', '--secret', 'Ad56bMSsbqPkjfMTVush7dBgYdLf27ul', '--endpoints', 'http://smithi196.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:06.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'M2KEYTP439P9KXXD', '--secret', 'Ad56bMSsbqPkjfMTVush7dBgYdLf27ul', '--endpoints', 'http://smithi196.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:06.625 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key M2KEYTP439P9KXXD --secret Ad56bMSsbqPkjfMTVush7dBgYdLf27ul --endpoints http://smithi196.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:26:06.721 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:26:06.721 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:26:06.780 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:26:06.774+0000 7fa27bee40c0 0 failed reading obj info from .rgw.root:zone_info.7ad4cc51-2d19-48f1-a21e-ef15faf62b94: (2) No such file or directory 2020-06-04T08:26:06.780 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:26:06.774+0000 7fa27bee40c0 0 WARNING: could not read zone params for zone id=7ad4cc51-2d19-48f1-a21e-ef15faf62b94 name=test-zone1 2020-06-04T08:26:06.794 INFO:teuthology.orchestra.run.smithi196.stdout:{"id":"d7df857b-1e19-4b4c-aa22-43c42d072194","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"M2KEYTP439P9KXXD","secret_key":"Ad56bMSsbqPkjfMTVush7dBgYdLf27ul"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","notif_pool":"test-zone2.rgw.log:notif"} 2020-06-04T08:26:06.795 INFO:teuthology.orchestra.run.smithi196:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2020-06-04T08:26:08.050 INFO:teuthology.orchestra.run.smithi196.stderr:pool 'test-zone2.rgw.buckets.data' created 2020-06-04T08:26:08.060 INFO:teuthology.orchestra.run.smithi196:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2020-06-04T08:26:09.056 INFO:teuthology.orchestra.run.smithi196.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2020-06-04T08:26:09.073 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:09.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:09.074 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:26:09.119 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:26:09.119 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:26:09.145 INFO:teuthology.orchestra.run.smithi196.stdout:{"id":"d7df857b-1e19-4b4c-aa22-43c42d072194","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"M2KEYTP439P9KXXD","secret_key":"Ad56bMSsbqPkjfMTVush7dBgYdLf27ul"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"test-zone2.rgw.log:notif"} 2020-06-04T08:26:09.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:09.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:09.146 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:26:09.181 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:26:09.181 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:26:10.278 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2020-06-04T08:26:17.244 INFO:teuthology.orchestra.run.smithi196.stderr:Sending period to new master zone 7ad4cc51-2d19-48f1-a21e-ef15faf62b94 2020-06-04T08:26:17.339 INFO:teuthology.orchestra.run.smithi196.stdout:{"id":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","epoch":2,"predecessor_uuid":"e080917d-3ba9-4cac-ad11-9299b0b52e3e","sync_status":[],"period_map":{"id":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","zones":[{"id":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","name":"test-zone1","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7df857b-1e19-4b4c-aa22-43c42d072194","name":"test-zone2","endpoints":["http://smithi196.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","val":3611466229},{"key":"d7df857b-1e19-4b4c-aa22-43c42d072194","val":443345077}]},"master_zonegroup":"test-zonegroup","master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","realm_name":"test-realm","realm_epoch":2} 2020-06-04T08:26:17.341 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2020-06-04T08:26:17.341 INFO:teuthology.orchestra.run:waiting for 300 2020-06-04T08:26:17.382 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:26:17.378+0000 7fcd75b90700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 15803) UID: 0 2020-06-04T08:26:17.383 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:26:17.378+0000 7fcd8d2210c0 -1 shutting down 2020-06-04T08:26:21.231 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2020-06-04T08:26:23.463 INFO:tasks.rgw.c2.client.0:Stopped 2020-06-04T08:26:23.464 INFO:tasks.rgw.c2.client.0:Restarting daemon 2020-06-04T08:26:23.464 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2020-06-04T08:26:23.485 INFO:tasks.rgw.c2.client.0:Started 2020-06-04T08:26:23.486 INFO:teuthology.orchestra.run.smithi196:> curl http://smithi196.front.sepia.ceph.com:8000 2020-06-04T08:26:23.504 INFO:teuthology.orchestra.run.smithi196.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-04T08:26:23.504 INFO:teuthology.orchestra.run.smithi196.stderr: Dload Upload Total Spent Left Speed 2020-06-04T08:26:23.506 INFO:teuthology.orchestra.run.smithi196.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi196.front.sepia.ceph.com port 8000: Connection refused 2020-06-04T08:26:23.506 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-06-04T08:26:24.508 INFO:teuthology.orchestra.run.smithi196:> curl http://smithi196.front.sepia.ceph.com:8000 2020-06-04T08:26:24.533 INFO:teuthology.orchestra.run.smithi196.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-04T08:26:24.533 INFO:teuthology.orchestra.run.smithi196.stderr: Dload Upload Total Spent Left Speed 2020-06-04T08:26:24.534 INFO:teuthology.orchestra.run.smithi196.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-06-04T08:26:24.534 INFO:teuthology.orchestra.run.smithi196.stdout:anonymous 2020-06-04T08:26:24.534 INFO:tasks.rgw_multisite:creating zone.. 2020-06-04T08:26:24.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'M2KEYTP439P9KXXD', '--secret', 'Ad56bMSsbqPkjfMTVush7dBgYdLf27ul', '--endpoints', 'http://smithi060.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:24.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'M2KEYTP439P9KXXD', '--secret', 'Ad56bMSsbqPkjfMTVush7dBgYdLf27ul', '--endpoints', 'http://smithi060.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:24.542 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key M2KEYTP439P9KXXD --secret Ad56bMSsbqPkjfMTVush7dBgYdLf27ul --endpoints http://smithi060.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:26:24.607 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:26:24.608 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:26:24.665 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:26:24.651+0000 7fce5caa10c0 0 failed reading obj info from .rgw.root:zone_info.d7df857b-1e19-4b4c-aa22-43c42d072194: (2) No such file or directory 2020-06-04T08:26:24.665 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:26:24.651+0000 7fce5caa10c0 0 WARNING: could not read zone params for zone id=d7df857b-1e19-4b4c-aa22-43c42d072194 name=test-zone2 2020-06-04T08:26:24.667 INFO:teuthology.orchestra.run.smithi060.stdout:{"id":"463a7035-f6f9-4bfc-a894-c421122fd949","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"M2KEYTP439P9KXXD","secret_key":"Ad56bMSsbqPkjfMTVush7dBgYdLf27ul"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","notif_pool":"test-zone3.rgw.log:notif"} 2020-06-04T08:26:24.668 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2020-06-04T08:26:25.581 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'test-zone3.rgw.buckets.data' created 2020-06-04T08:26:25.611 INFO:teuthology.orchestra.run.smithi060:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2020-06-04T08:26:26.618 INFO:teuthology.orchestra.run.smithi060.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2020-06-04T08:26:26.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:26.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:26.632 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:26:26.687 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:26:26.688 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:26:26.717 INFO:teuthology.orchestra.run.smithi060.stdout:{"id":"463a7035-f6f9-4bfc-a894-c421122fd949","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"M2KEYTP439P9KXXD","secret_key":"Ad56bMSsbqPkjfMTVush7dBgYdLf27ul"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":"","notif_pool":"test-zone3.rgw.log:notif"} 2020-06-04T08:26:26.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:26.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:26.719 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:26:26.762 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:26:26.763 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:26:26.938 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2020-06-04T08:26:34.807 INFO:teuthology.orchestra.run.smithi060.stderr:Sending period to new master zone 7ad4cc51-2d19-48f1-a21e-ef15faf62b94 2020-06-04T08:26:34.903 INFO:teuthology.orchestra.run.smithi060.stdout:{"id":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","epoch":3,"predecessor_uuid":"e080917d-3ba9-4cac-ad11-9299b0b52e3e","sync_status":[],"period_map":{"id":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","zones":[{"id":"463a7035-f6f9-4bfc-a894-c421122fd949","name":"test-zone3","endpoints":["http://smithi060.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","name":"test-zone1","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7df857b-1e19-4b4c-aa22-43c42d072194","name":"test-zone2","endpoints":["http://smithi196.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"463a7035-f6f9-4bfc-a894-c421122fd949","val":2270195151},{"key":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","val":3611466229},{"key":"d7df857b-1e19-4b4c-aa22-43c42d072194","val":443345077}]},"master_zonegroup":"test-zonegroup","master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","realm_name":"test-realm","realm_epoch":2} 2020-06-04T08:26:34.905 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-06-04T08:26:34.905 INFO:teuthology.orchestra.run:waiting for 300 2020-06-04T08:26:34.946 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-06-04T08:26:34.943+0000 7f4f4effd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 15800) UID: 0 2020-06-04T08:26:34.946 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-06-04T08:26:34.943+0000 7f4f728290c0 -1 shutting down 2020-06-04T08:26:38.716 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-06-04T08:26:41.011 INFO:tasks.rgw.c1.client.1:Stopped 2020-06-04T08:26:41.012 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-06-04T08:26:41.013 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-06-04T08:26:41.017 INFO:tasks.rgw.c1.client.1:Started 2020-06-04T08:26:41.018 INFO:teuthology.orchestra.run.smithi060:> curl http://smithi060.front.sepia.ceph.com:8001 2020-06-04T08:26:41.053 INFO:teuthology.orchestra.run.smithi060.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-04T08:26:41.055 INFO:teuthology.orchestra.run.smithi060.stderr: Dload Upload Total Spent Left Speed 2020-06-04T08:26:41.056 INFO:teuthology.orchestra.run.smithi060.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi060.front.sepia.ceph.com port 8001: Connection refused 2020-06-04T08:26:41.056 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-06-04T08:26:42.058 INFO:teuthology.orchestra.run.smithi060:> curl http://smithi060.front.sepia.ceph.com:8001 2020-06-04T08:26:42.075 INFO:teuthology.orchestra.run.smithi060.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-04T08:26:42.075 INFO:teuthology.orchestra.run.smithi060.stderr: Dload Upload Total Spent Left Speed 2020-06-04T08:26:42.080 INFO:teuthology.orchestra.run.smithi060.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2020-06-04T08:26:42.081 INFO:teuthology.orchestra.run.smithi060.stdout:anonymous 2020-06-04T08:26:42.081 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2020-06-04T08:26:42.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi060.front.sepia.ceph.com:8000', '--access-key', 'M2KEYTP439P9KXXD', '--secret', 'Ad56bMSsbqPkjfMTVush7dBgYdLf27ul', '--url', 'http://smithi060.front.sepia.ceph.com:8000', '--access-key', 'M2KEYTP439P9KXXD', '--secret', 'Ad56bMSsbqPkjfMTVush7dBgYdLf27ul', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:42.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi060.front.sepia.ceph.com:8000', '--access-key', 'M2KEYTP439P9KXXD', '--secret', 'Ad56bMSsbqPkjfMTVush7dBgYdLf27ul', '--url', 'http://smithi060.front.sepia.ceph.com:8000', '--access-key', 'M2KEYTP439P9KXXD', '--secret', 'Ad56bMSsbqPkjfMTVush7dBgYdLf27ul', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:42.083 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi060.front.sepia.ceph.com:8000 --access-key M2KEYTP439P9KXXD --secret Ad56bMSsbqPkjfMTVush7dBgYdLf27ul --url http://smithi060.front.sepia.ceph.com:8000 --access-key M2KEYTP439P9KXXD --secret Ad56bMSsbqPkjfMTVush7dBgYdLf27ul --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:26:42.126 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:26:42.127 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:26:42.166 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:26:42.162+0000 7f83ee7fc700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.9c559937-87e1-4867-81f3-e687b7a20f8b.control err=-107 2020-06-04T08:26:42.190 INFO:teuthology.orchestra.run.smithi196.stdout:{"id":"9c559937-87e1-4867-81f3-e687b7a20f8b","name":"test-realm","current_period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","epoch":2} 2020-06-04T08:26:42.191 INFO:tasks.rgw_multisite:creating zone.. 2020-06-04T08:26:42.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'M2KEYTP439P9KXXD', '--secret', 'Ad56bMSsbqPkjfMTVush7dBgYdLf27ul', '--endpoints', 'http://smithi196.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:42.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'M2KEYTP439P9KXXD', '--secret', 'Ad56bMSsbqPkjfMTVush7dBgYdLf27ul', '--endpoints', 'http://smithi196.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:42.193 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key M2KEYTP439P9KXXD --secret Ad56bMSsbqPkjfMTVush7dBgYdLf27ul --endpoints http://smithi196.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:26:42.285 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:26:42.285 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:26:42.351 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:26:42.346+0000 7f9f93a3e0c0 0 failed reading obj info from .rgw.root:zone_info.463a7035-f6f9-4bfc-a894-c421122fd949: (2) No such file or directory 2020-06-04T08:26:42.352 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:26:42.346+0000 7f9f93a3e0c0 0 WARNING: could not read zone params for zone id=463a7035-f6f9-4bfc-a894-c421122fd949 name=test-zone3 2020-06-04T08:26:42.352 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:26:42.346+0000 7f9f93a3e0c0 0 failed reading obj info from .rgw.root:zone_info.7ad4cc51-2d19-48f1-a21e-ef15faf62b94: (2) No such file or directory 2020-06-04T08:26:42.352 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:26:42.346+0000 7f9f93a3e0c0 0 WARNING: could not read zone params for zone id=7ad4cc51-2d19-48f1-a21e-ef15faf62b94 name=test-zone1 2020-06-04T08:26:42.363 INFO:teuthology.orchestra.run.smithi196.stdout:{"id":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"M2KEYTP439P9KXXD","secret_key":"Ad56bMSsbqPkjfMTVush7dBgYdLf27ul"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","notif_pool":"test-zone4.rgw.log:notif"} 2020-06-04T08:26:42.363 INFO:teuthology.orchestra.run.smithi196:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2020-06-04T08:26:42.953 INFO:teuthology.orchestra.run.smithi196.stderr:pool 'test-zone4.rgw.buckets.data' created 2020-06-04T08:26:42.973 INFO:teuthology.orchestra.run.smithi196:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2020-06-04T08:26:43.970 INFO:teuthology.orchestra.run.smithi196.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2020-06-04T08:26:43.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:43.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:43.988 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:26:44.046 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:26:44.047 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:26:44.069 INFO:teuthology.orchestra.run.smithi196.stdout:{"id":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"M2KEYTP439P9KXXD","secret_key":"Ad56bMSsbqPkjfMTVush7dBgYdLf27ul"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":"","notif_pool":"test-zone4.rgw.log:notif"} 2020-06-04T08:26:44.069 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:44.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:44.070 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:26:44.106 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:26:44.107 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:26:44.421 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-06-04T08:26:52.164 INFO:teuthology.orchestra.run.smithi196.stderr:Sending period to new master zone 7ad4cc51-2d19-48f1-a21e-ef15faf62b94 2020-06-04T08:26:52.331 INFO:teuthology.orchestra.run.smithi196.stdout:{"id":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","epoch":4,"predecessor_uuid":"e080917d-3ba9-4cac-ad11-9299b0b52e3e","sync_status":[],"period_map":{"id":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","zones":[{"id":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","name":"test-zone4","endpoints":["http://smithi196.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"463a7035-f6f9-4bfc-a894-c421122fd949","name":"test-zone3","endpoints":["http://smithi060.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","name":"test-zone1","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7df857b-1e19-4b4c-aa22-43c42d072194","name":"test-zone2","endpoints":["http://smithi196.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","val":2528435237},{"key":"463a7035-f6f9-4bfc-a894-c421122fd949","val":2270195151},{"key":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","val":3611466229},{"key":"d7df857b-1e19-4b4c-aa22-43c42d072194","val":443345077}]},"master_zonegroup":"test-zonegroup","master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","realm_name":"test-realm","realm_epoch":2} 2020-06-04T08:26:52.332 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2020-06-04T08:26:52.333 INFO:teuthology.orchestra.run:waiting for 300 2020-06-04T08:26:52.374 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:26:52.370+0000 7fe023fff700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 15809) UID: 0 2020-06-04T08:26:52.374 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:26:52.370+0000 7fe0476b10c0 -1 shutting down 2020-06-04T08:26:55.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2020-06-04T08:26:58.435 INFO:tasks.rgw.c2.client.1:Stopped 2020-06-04T08:26:58.436 INFO:tasks.rgw.c2.client.1:Restarting daemon 2020-06-04T08:26:58.437 INFO:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2020-06-04T08:26:58.440 INFO:tasks.rgw.c2.client.1:Started 2020-06-04T08:26:58.440 INFO:teuthology.orchestra.run.smithi196:> curl http://smithi196.front.sepia.ceph.com:8001 2020-06-04T08:26:58.467 INFO:teuthology.orchestra.run.smithi196.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-04T08:26:58.468 INFO:teuthology.orchestra.run.smithi196.stderr: Dload Upload Total Spent Left Speed 2020-06-04T08:26:58.468 INFO:teuthology.orchestra.run.smithi196.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi196.front.sepia.ceph.com port 8001: Connection refused 2020-06-04T08:26:58.468 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-06-04T08:26:59.469 INFO:teuthology.orchestra.run.smithi196:> curl http://smithi196.front.sepia.ceph.com:8001 2020-06-04T08:26:59.486 INFO:teuthology.orchestra.run.smithi196.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-04T08:26:59.486 INFO:teuthology.orchestra.run.smithi196.stderr: Dload Upload Total Spent Left Speed 2020-06-04T08:26:59.490 INFO:teuthology.orchestra.run.smithi196.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 33500 0 --:--:-- --:--:-- --:--:-- 33500 2020-06-04T08:26:59.492 INFO:teuthology.orchestra.run.smithi196.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-005ed8b053-115e-test-zone4","HostId":"115e-test-zone4-test-zonegroup"} 2020-06-04T08:26:59.492 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2020-06-04T08:26:59.493 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2020-06-04T08:26:59.599 INFO:tasks.rgw_multisite_tests:creating test user.. 2020-06-04T08:26:59.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:59.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:26:59.601 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:26:59.647 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:26:59.648 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:26:59.839 INFO:teuthology.orchestra.run.smithi060.stdout:{"user_id":"rgw-multisite-test-user","display_name":"Multisite Test User","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"rgw-multisite-test-user","access_key":"038Q7JN7FVHJ9K8KVTPF","secret_key":"MGwMH1rUFkoByQhe5lhjKzcBBSOgsxKgHpLFPxAi"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2020-06-04T08:26:59.870 INFO:rgw_multi.tests:meta checkpoint 2020-06-04T08:26:59.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:26:59.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:26:59.872 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:26:59.954 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:26:59.954 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:27:00.134 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=838ed6b2-fa2f-4917-a5c1-6c82bb405cda 2020-06-04T08:27:00.168 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259219.830770_41.1","last_update":"2020-06-04T08:26:59.830770Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:27:00.169 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:27:00.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:00.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:00.171 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:27:00.209 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:27:00.209 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:27:00.413 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:27:00.414 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259219.830770_41.1 target= 2020-06-04T08:27:01.105 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2020-06-04T08:27:05.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:05.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:05.418 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:27:05.461 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:27:05.461 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:27:05.674 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:27:05.675 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259219.830770_41.1 target= 2020-06-04T08:27:10.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:10.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:10.679 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:27:10.718 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:27:10.718 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:27:10.930 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:27:10.931 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259219.830770_41.1 target= 2020-06-04T08:27:15.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:15.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:15.934 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:27:15.982 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:27:15.982 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:27:16.148 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:27:16.149 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259219.830770_41.1 target= 2020-06-04T08:27:21.150 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:21.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:21.152 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:27:21.198 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:27:21.198 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:27:21.408 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:27:21.409 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259219.830770_41.1 target= 2020-06-04T08:27:26.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:26.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:26.411 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:27:26.454 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:27:26.454 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:27:26.661 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:27:26.661 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259219.830770_41.1 target= 2020-06-04T08:27:31.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:31.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:31.668 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:27:31.711 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:27:31.711 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:27:31.917 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:27:31.918 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259219.830770_41.1 target= 2020-06-04T08:27:36.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:36.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:36.929 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:27:36.978 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:27:36.979 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:27:37.183 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:27:37.183 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259219.830770_41.1 target= 2020-06-04T08:27:42.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:42.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:42.186 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:27:42.241 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:27:42.241 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:27:42.425 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:27:42.426 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259219.830770_41.1 target= 2020-06-04T08:27:47.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:47.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:47.428 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:27:47.471 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:27:47.471 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:27:47.687 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:27:47.687 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259219.830770_41.1 target= 2020-06-04T08:27:52.689 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:52.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:52.691 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:27:52.733 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:27:52.734 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:27:52.933 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:27:52.934 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259219.830770_41.1 target= 2020-06-04T08:27:57.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:57.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:27:57.936 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:27:57.977 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:27:57.977 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:27:58.198 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:27:58.199 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259219.830770_41.1 target= 2020-06-04T08:28:03.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:03.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:03.202 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:28:03.244 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:28:03.245 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:28:03.453 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:28:03.454 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259219.830770_41.1 target= 2020-06-04T08:28:08.455 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:08.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:08.457 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:28:08.497 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:28:08.497 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:28:08.696 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:28:08.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259219.830770_41.1 target= 2020-06-04T08:28:13.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:13.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:13.700 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:28:13.743 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:28:13.743 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:28:13.952 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:28:13.953 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259219.830770_41.1 target= 2020-06-04T08:28:18.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:18.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:18.957 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:28:19.004 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:28:19.006 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:28:19.203 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:28:19.203 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259219.830770_41.1 target= 2020-06-04T08:28:24.205 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:24.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:24.209 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:28:24.252 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:28:24.253 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:28:24.464 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:28:24.465 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259219.830770_41.1 target= 2020-06-04T08:28:29.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:29.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:29.469 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:28:29.508 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:28:29.509 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:28:29.712 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:28:29.712 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259219.830770_41.1 target= 2020-06-04T08:28:34.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:34.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:34.717 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:28:34.761 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:28:34.762 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:28:34.972 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:28:34.973 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259219.830770_41.1 target= 2020-06-04T08:28:39.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:39.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:39.980 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:28:40.034 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:28:40.034 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:28:40.226 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:28:40.227 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259219.830770_41.1 target= 2020-06-04T08:28:45.228 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:45.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:45.230 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:28:45.271 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:28:45.271 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:28:45.477 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:28:45.478 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259219.830770_41.1 target= 2020-06-04T08:28:50.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:50.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:50.481 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:28:50.526 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:28:50.526 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:28:50.730 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:28:50.731 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259219.830770_41.1 target= 2020-06-04T08:28:55.732 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:55.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:55.734 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:28:55.775 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:28:55.775 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:28:55.986 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259219.830770_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:26:59.830770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:28:55.989 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:28:55.990 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:28:55.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:55.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:56.005 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:28:56.050 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:28:56.050 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:28:56.268 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259219.830770_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:26:59.830770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:28:56.269 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:28:56.270 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:28:56.270 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:56.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:56.271 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:28:56.313 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:28:56.313 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:28:56.576 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259219.830770_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:26:59.830770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:28:56.577 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:28:56.577 INFO:rgw_multi.tests:meta checkpoint 2020-06-04T08:28:56.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:56.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:56.578 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:28:56.621 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:28:56.622 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:28:56.795 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=838ed6b2-fa2f-4917-a5c1-6c82bb405cda 2020-06-04T08:28:56.829 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259219.830770_41.1","last_update":"2020-06-04T08:26:59.830770Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:28:56.830 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:28:56.830 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:56.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:56.831 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:28:56.873 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:28:56.874 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:28:57.088 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259219.830770_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:26:59.830770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:28:57.089 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:28:57.089 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:28:57.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:57.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:57.091 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:28:57.133 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:28:57.134 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:28:57.350 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259219.830770_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:26:59.830770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:28:57.351 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:28:57.352 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:28:57.352 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:57.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:28:57.354 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:28:57.394 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:28:57.395 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:28:57.857 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259219.830770_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:26:59.830770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:28:57.858 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:28:57.861 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2020-06-04T08:28:57.874 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-1 2020-06-04T08:29:01.748 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gpxthc-2 2020-06-04T08:29:05.159 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gpxthc-3 2020-06-04T08:29:05.225 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:05.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:05.228 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:05.266 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:05.267 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:05.438 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=838ed6b2-fa2f-4917-a5c1-6c82bb405cda 2020-06-04T08:29:05.468 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259219.830770_41.1","last_update":"2020-06-04T08:26:59.830770Z"},{"marker":"1_1591259345.209069_45.1","last_update":"2020-06-04T08:29:05.209069Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:29:05.469 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:29:05.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:05.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:05.472 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:05.512 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:05.512 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:05.726 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259219.830770_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:26:59.830770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259345.209069_45.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:05.209069Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:29:05.727 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:29:05.728 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:29:05.729 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:05.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:05.732 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:05.792 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:05.793 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:06.001 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259219.830770_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:26:59.830770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259345.209069_45.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:05.209069Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:29:06.002 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:29:06.002 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:29:06.006 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:06.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:06.008 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:06.046 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:06.047 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:06.313 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259219.830770_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:26:59.830770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259345.209069_45.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:05.209069Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:29:06.313 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:29:06.368 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2020-06-04T08:29:06.370 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-4 2020-06-04T08:29:06.396 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gpxthc-5 2020-06-04T08:29:06.445 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gpxthc-6 2020-06-04T08:29:06.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:06.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:06.491 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:06.531 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:06.531 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:06.687 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=838ed6b2-fa2f-4917-a5c1-6c82bb405cda 2020-06-04T08:29:06.713 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259219.830770_41.1","last_update":"2020-06-04T08:26:59.830770Z"},{"marker":"1_1591259346.473034_57.1","last_update":"2020-06-04T08:29:06.473034Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:29:06.714 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:29:06.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:06.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:06.716 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:06.762 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:06.763 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:06.959 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259219.830770_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:26:59.830770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259346.473034_57.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:06.473034Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:29:06.960 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:29:06.961 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:29:06.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:06.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:06.964 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:07.007 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:07.009 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:07.232 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259219.830770_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:26:59.830770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259346.473034_57.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:06.473034Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:29:07.233 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:29:07.234 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:29:07.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:07.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:07.236 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:07.278 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:07.278 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:07.543 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259219.830770_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:26:59.830770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259346.473034_57.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:06.473034Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:29:07.544 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:29:08.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:08.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:08.307 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:08.340 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:08.341 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:08.520 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=838ed6b2-fa2f-4917-a5c1-6c82bb405cda 2020-06-04T08:29:08.553 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259219.830770_41.1","last_update":"2020-06-04T08:26:59.830770Z"},{"marker":"1_1591259348.175359_93.1","last_update":"2020-06-04T08:29:08.175359Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:29:08.554 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:29:08.555 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:08.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:08.556 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:08.595 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:08.596 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:08.794 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259219.830770_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:26:59.830770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259348.175359_93.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:08.175359Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:29:08.794 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:29:08.795 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:29:08.796 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:08.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:08.797 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:08.838 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:08.839 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:09.052 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259219.830770_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:26:59.830770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259348.175359_93.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:08.175359Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:29:09.053 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:29:09.054 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:29:09.055 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:09.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:09.057 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:09.099 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:09.099 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:09.352 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259219.830770_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:26:59.830770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259348.175359_93.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:08.175359Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:29:09.353 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:29:09.430 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2020-06-04T08:29:09.431 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-7 2020-06-04T08:29:09.458 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gpxthc-8 2020-06-04T08:29:09.526 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gpxthc-9 2020-06-04T08:29:09.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:09.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:09.578 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:09.614 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:09.615 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:09.783 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=838ed6b2-fa2f-4917-a5c1-6c82bb405cda 2020-06-04T08:29:09.815 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259219.830770_41.1","last_update":"2020-06-04T08:26:59.830770Z"},{"marker":"1_1591259349.557874_105.1","last_update":"2020-06-04T08:29:09.557874Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:29:09.817 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:29:09.818 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:09.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:09.820 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:09.858 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:09.859 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:10.055 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259219.830770_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:26:59.830770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259349.557874_105.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:09.557874Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:29:10.119 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:29:10.120 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:29:10.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:10.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:10.123 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:10.164 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:10.165 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:10.391 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259219.830770_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:26:59.830770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259349.557874_105.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:09.557874Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:29:10.392 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:29:10.393 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:29:10.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:10.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:10.403 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:10.446 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:10.447 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:10.712 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259219.830770_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:26:59.830770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259349.557874_105.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:09.557874Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:29:10.713 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:29:11.013 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:11.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:11.015 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:11.050 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:11.050 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:11.234 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=838ed6b2-fa2f-4917-a5c1-6c82bb405cda 2020-06-04T08:29:11.270 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259219.830770_41.1","last_update":"2020-06-04T08:26:59.830770Z"},{"marker":"1_1591259350.985454_111.1","last_update":"2020-06-04T08:29:10.985454Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:29:11.271 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:29:11.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:11.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:11.273 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:11.325 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:11.325 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:11.526 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259219.830770_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:26:59.830770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:29:11.527 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:29:11.528 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:29:11.531 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:11.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:11.533 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:11.576 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:11.576 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:11.783 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259219.830770_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:26:59.830770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:29:11.783 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:29:11.784 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:29:11.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:11.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:11.791 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:11.837 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:11.838 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:12.109 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259219.830770_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:26:59.830770Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:29:12.110 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:29:12.159 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2020-06-04T08:29:12.161 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-10 2020-06-04T08:29:12.195 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gpxthc-11 2020-06-04T08:29:12.247 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gpxthc-12 2020-06-04T08:29:12.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:12.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:12.968 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:13.017 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:13.017 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:13.194 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=838ed6b2-fa2f-4917-a5c1-6c82bb405cda 2020-06-04T08:29:13.222 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259352.279189_60.1","last_update":"2020-06-04T08:29:12.279189Z"},{"marker":"1_1591259350.985454_111.1","last_update":"2020-06-04T08:29:10.985454Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:29:13.223 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:29:13.224 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:13.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:13.225 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:13.268 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:13.268 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:13.440 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259352.273166_58.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:29:12.273166Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:29:13.442 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259352.279189_60.1 target=1_1591259352.273166_58.1 2020-06-04T08:29:18.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:18.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:18.448 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:18.506 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:18.507 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:18.719 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259352.279189_60.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:29:12.279189Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:29:18.719 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:29:18.720 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:29:18.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:18.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:18.722 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:18.780 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:18.781 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:18.978 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259352.279189_60.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:29:12.279189Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:29:18.979 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:29:18.980 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:29:18.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:18.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:18.988 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:19.035 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:19.036 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:19.413 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259352.279189_60.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:29:12.279189Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:29:19.414 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:29:19.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:19.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:19.417 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:19.463 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:19.464 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:19.679 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.13.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.12.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.12.5"},{"key":8,"val":"00000000001.6.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:29:19.680 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-10 2020-06-04T08:29:19.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:19.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:19.682 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-10 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:19.734 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:19.734 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:19.959 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-06-04T08:29:12.559597Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.5","timestamp":"2020-06-04T08:29:12.559535Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.5","timestamp":"2020-06-04T08:29:12.516159Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5","timestamp":"2020-06-04T08:29:12.555481Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:29:19.960 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-10 2020-06-04T08:29:19.962 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-10 zones={test-zone2, test-zone1} 2020-06-04T08:29:19.991 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:29:20.018 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-04T08:29:20.019 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-04T08:29:20.033 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-04T08:29:20.034 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:20.035 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:29:20.052 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-04T08:29:20.053 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-04T08:29:20.054 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-04T08:29:20.066 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:20.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-04T08:29:20.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-04T08:29:20.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-04T08:29:20.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-04T08:29:20.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-04T08:29:20.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-04T08:29:20.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-04T08:29:20.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-04T08:29:20.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-04T08:29:21.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:21.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:21.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:21.341 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-10 zones={test-zone2, test-zone1} 2020-06-04T08:29:21.342 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:21.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:21.344 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:21.397 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:21.397 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:21.611 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.13.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.12.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.12.5"},{"key":8,"val":"00000000001.6.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:29:21.612 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-10 2020-06-04T08:29:21.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:21.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:21.615 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:21.699 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:21.932 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-06-04T08:29:12.553559Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.5","timestamp":"2020-06-04T08:29:12.553595Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.5","timestamp":"2020-06-04T08:29:12.512412Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5","timestamp":"2020-06-04T08:29:12.553611Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:29:21.933 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-10 2020-06-04T08:29:21.933 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-10 zones={test-zone3, test-zone1} 2020-06-04T08:29:21.950 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:29:21.960 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-04T08:29:21.961 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-04T08:29:21.963 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-04T08:29:21.964 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:21.965 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:29:21.981 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-04T08:29:21.982 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-04T08:29:21.982 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-04T08:29:21.982 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:22.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-04T08:29:22.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-04T08:29:22.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-04T08:29:22.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-04T08:29:22.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-04T08:29:22.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-04T08:29:22.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-04T08:29:22.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-04T08:29:22.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-04T08:29:22.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:22.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:23.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:23.201 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-10 zones={test-zone3, test-zone1} 2020-06-04T08:29:23.202 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:23.202 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:23.203 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:23.243 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:23.243 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:23.463 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.13.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.12.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.12.5"},{"key":8,"val":"00000000001.6.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:29:23.463 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-10 2020-06-04T08:29:23.464 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:23.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:23.465 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-10 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:23.510 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:23.510 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:23.780 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-06-04T08:29:12.328442Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.12.5","timestamp":"2020-06-04T08:29:12.376023Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.12.5","timestamp":"2020-06-04T08:29:12.474174Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.6.5","timestamp":"2020-06-04T08:29:12.425702Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:29:23.781 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-10 2020-06-04T08:29:23.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:23.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:23.784 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:23.847 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:23.848 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:24.043 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.14.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.8.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.13.5"},{"key":8,"val":"00000000001.8.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:29:24.044 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-11 2020-06-04T08:29:24.045 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:24.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:24.046 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:24.094 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:24.095 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:24.328 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.14.5","timestamp":"2020-06-04T08:29:12.737439Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5","timestamp":"2020-06-04T08:29:12.695424Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-06-04T08:29:12.973452Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5","timestamp":"2020-06-04T08:29:12.695447Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:29:24.329 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-11 2020-06-04T08:29:24.331 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-11 zones={test-zone1, test-zone2} 2020-06-04T08:29:24.345 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:29:24.369 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-04T08:29:24.371 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-04T08:29:24.372 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-04T08:29:24.372 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:24.374 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:29:24.390 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-04T08:29:24.392 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-04T08:29:24.392 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-04T08:29:24.393 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:24.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-04T08:29:24.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-04T08:29:24.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-04T08:29:24.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-04T08:29:24.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-04T08:29:24.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-04T08:29:25.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-04T08:29:25.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-04T08:29:25.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-04T08:29:25.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:25.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:25.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:25.684 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-11 zones={test-zone1, test-zone2} 2020-06-04T08:29:25.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:25.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:25.689 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:25.734 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:25.735 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:25.944 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.14.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.8.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.13.5"},{"key":8,"val":"00000000001.8.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:29:25.945 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-11 2020-06-04T08:29:25.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:25.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:25.953 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:26.000 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:26.001 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:26.246 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.14.5","timestamp":"2020-06-04T08:29:12.729516Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5","timestamp":"2020-06-04T08:29:12.729492Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-06-04T08:29:12.965389Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5","timestamp":"2020-06-04T08:29:12.729545Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:29:26.247 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-11 2020-06-04T08:29:26.248 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-11 zones={test-zone3, test-zone2} 2020-06-04T08:29:26.271 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:29:26.295 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-04T08:29:26.307 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-04T08:29:26.308 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-04T08:29:26.308 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:26.309 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:29:26.335 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-04T08:29:26.336 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-04T08:29:26.336 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-04T08:29:26.339 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:26.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-04T08:29:26.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-04T08:29:26.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-04T08:29:26.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-04T08:29:26.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-04T08:29:26.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-04T08:29:27.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-04T08:29:27.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-04T08:29:27.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-04T08:29:27.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:27.422 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:27.541 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:27.660 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-11 zones={test-zone3, test-zone2} 2020-06-04T08:29:27.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:27.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:27.663 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:27.717 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:27.717 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:27.940 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.6.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.13.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.8.5"},{"key":8,"val":"00000000001.8.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:29:27.941 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-12 2020-06-04T08:29:27.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:27.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:27.944 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:28.030 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:28.031 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:28.267 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5","timestamp":"2020-06-04T08:29:12.813451Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-06-04T08:29:12.978073Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5","timestamp":"2020-06-04T08:29:12.978039Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5","timestamp":"2020-06-04T08:29:13.017431Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:29:28.268 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-12 2020-06-04T08:29:28.269 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-12 zones={test-zone1, test-zone3} 2020-06-04T08:29:28.288 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:29:28.312 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-04T08:29:28.313 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-04T08:29:28.319 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-04T08:29:28.320 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:28.321 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:29:28.337 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-04T08:29:28.338 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-04T08:29:28.339 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-04T08:29:28.339 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:28.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-04T08:29:28.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-04T08:29:28.573 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-04T08:29:28.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-04T08:29:28.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-04T08:29:28.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-04T08:29:29.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-04T08:29:29.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-04T08:29:29.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-04T08:29:29.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:29.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:29.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:29.665 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-12 zones={test-zone1, test-zone3} 2020-06-04T08:29:29.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:29.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:29.668 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:29.706 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:29.707 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:29.924 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.6.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.13.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.8.5"},{"key":8,"val":"00000000001.8.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:29:29.925 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-12 2020-06-04T08:29:29.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:29.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:29.926 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-12 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:29.981 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:29.981 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:30.201 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5","timestamp":"2020-06-04T08:29:12.811442Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5","timestamp":"2020-06-04T08:29:13.015489Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5","timestamp":"2020-06-04T08:29:13.015544Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5","timestamp":"2020-06-04T08:29:12.975247Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:29:30.202 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-12 2020-06-04T08:29:30.203 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-12 zones={test-zone2, test-zone3} 2020-06-04T08:29:30.217 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:29:30.231 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-04T08:29:30.232 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-04T08:29:30.233 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-04T08:29:30.234 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:30.235 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:29:30.248 DEBUG:tasks.rgw_multi.zone_rados:o=& 2020-06-04T08:29:30.249 DEBUG:tasks.rgw_multi.zone_rados:o=: 2020-06-04T08:29:30.249 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2020-06-04T08:29:30.250 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:30.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-04T08:29:30.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-04T08:29:30.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2020-06-04T08:29:30.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-04T08:29:30.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-04T08:29:30.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2020-06-04T08:29:30.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-04T08:29:31.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-04T08:29:31.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2020-06-04T08:29:31.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:31.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:31.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:31.602 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-12 zones={test-zone2, test-zone3} 2020-06-04T08:29:31.604 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... ok 2020-06-04T08:29:31.605 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-13 2020-06-04T08:29:31.633 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gpxthc-14 2020-06-04T08:29:31.685 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gpxthc-15 2020-06-04T08:29:31.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:31.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:31.908 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:31.950 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:31.951 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:32.106 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=838ed6b2-fa2f-4917-a5c1-6c82bb405cda 2020-06-04T08:29:32.131 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259371.724712_78.1","last_update":"2020-06-04T08:29:31.724712Z"},{"marker":"1_1591259350.985454_111.1","last_update":"2020-06-04T08:29:10.985454Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:29:32.131 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:29:32.132 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:32.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:32.134 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:32.181 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:32.182 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:32.358 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259371.724712_78.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:29:31.724712Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:29:32.359 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:29:32.360 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:29:32.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:32.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:32.364 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:32.411 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:32.411 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:32.611 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259371.724712_78.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:29:31.724712Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:29:32.612 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:29:32.613 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:29:32.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:32.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:32.618 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:32.658 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:32.659 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:32.926 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259371.724712_78.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:29:31.724712Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:29:32.927 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:29:32.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:32.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:32.929 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:32.973 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:32.974 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:33.190 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.17.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:29:33.190 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-13 2020-06-04T08:29:33.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:33.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:33.192 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:33.238 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:33.239 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:33.469 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.17.5","timestamp":"2020-06-04T08:29:31.959434Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:29:33.470 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-13 2020-06-04T08:29:33.471 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-13 zones={test-zone2, test-zone1} 2020-06-04T08:29:33.489 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:29:33.542 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:33.543 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:29:33.555 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:33.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:33.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:33.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:34.001 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-13 zones={test-zone2, test-zone1} 2020-06-04T08:29:34.002 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:34.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:34.003 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:34.042 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:34.043 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:34.260 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.17.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:29:34.260 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-13 2020-06-04T08:29:34.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:34.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:34.262 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:34.358 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:34.359 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:34.603 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.17.5","timestamp":"2020-06-04T08:29:31.961466Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:29:34.603 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-13 2020-06-04T08:29:34.604 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-13 zones={test-zone3, test-zone1} 2020-06-04T08:29:34.614 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:29:34.664 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:34.665 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:29:34.676 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:34.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:34.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:34.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:35.064 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-13 zones={test-zone3, test-zone1} 2020-06-04T08:29:35.065 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:35.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:35.069 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:35.107 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:35.107 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:35.330 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.17.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:29:35.330 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-13 2020-06-04T08:29:35.331 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:35.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:35.342 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:35.392 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:35.392 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:35.667 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.17.5","timestamp":"2020-06-04T08:29:31.759602Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:29:35.668 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-13 2020-06-04T08:29:35.668 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:35.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:35.669 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:35.756 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:35.756 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:35.967 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.14.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:29:35.968 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-14 2020-06-04T08:29:35.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:35.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:35.975 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:36.022 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:36.023 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:36.262 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.14.5","timestamp":"2020-06-04T08:29:31.941462Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:29:36.263 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-14 2020-06-04T08:29:36.264 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-14 zones={test-zone1, test-zone2} 2020-06-04T08:29:36.276 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:29:36.328 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:36.329 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:29:36.338 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:36.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:36.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:36.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:36.723 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-14 zones={test-zone1, test-zone2} 2020-06-04T08:29:36.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:36.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:36.727 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:36.770 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:36.771 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:36.974 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.14.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:29:36.975 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-14 2020-06-04T08:29:36.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:36.976 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:36.976 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:37.023 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:37.024 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:37.269 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.14.5","timestamp":"2020-06-04T08:29:31.937446Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:29:37.269 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-14 2020-06-04T08:29:37.270 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-14 zones={test-zone3, test-zone2} 2020-06-04T08:29:37.289 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:29:37.342 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:37.344 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:29:37.353 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:37.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:37.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:37.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:37.697 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-14 zones={test-zone3, test-zone2} 2020-06-04T08:29:37.698 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:37.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:37.699 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:37.742 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:37.742 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:37.963 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.8.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:29:37.964 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-15 2020-06-04T08:29:37.964 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:37.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:37.965 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:38.050 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:38.050 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:38.302 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5","timestamp":"2020-06-04T08:29:32.029402Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:29:38.303 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-15 2020-06-04T08:29:38.304 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-15 zones={test-zone1, test-zone3} 2020-06-04T08:29:38.317 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:29:38.369 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:38.370 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:29:38.389 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:38.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:38.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:38.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:38.868 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-15 zones={test-zone1, test-zone3} 2020-06-04T08:29:38.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:38.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:38.871 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:38.923 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:38.924 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:39.146 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.8.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:29:39.147 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-15 2020-06-04T08:29:39.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:39.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:39.150 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-15 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:39.198 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:39.198 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:39.431 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5","timestamp":"2020-06-04T08:29:32.023472Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:29:39.432 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-15 2020-06-04T08:29:39.433 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-15 zones={test-zone2, test-zone3} 2020-06-04T08:29:39.446 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:29:39.494 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:39.495 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:29:39.507 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2020-06-04T08:29:39.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:39.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:39.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2020-06-04T08:29:39.881 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-15 zones={test-zone2, test-zone3} 2020-06-04T08:29:39.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:39.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:39.912 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:39.954 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:39.955 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:40.175 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.19.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:29:40.175 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-13 2020-06-04T08:29:40.176 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:40.177 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:40.178 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:40.236 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:40.237 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:40.466 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.17.5","timestamp":"2020-06-04T08:29:31.959434Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:29:40.467 WARNING:rgw_multi.tests:bucket gpxthc-13 zone test-zone2 behind zone test-zone1: shard=1 master=00000000002.19.5 target=00000000001.17.5 2020-06-04T08:29:45.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:45.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:45.469 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:45.514 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:45.514 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:45.744 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.17.5","timestamp":"2020-06-04T08:29:31.959434Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:29:45.745 WARNING:rgw_multi.tests:bucket gpxthc-13 zone test-zone2 behind zone test-zone1: shard=1 master=00000000002.19.5 target=00000000001.17.5 2020-06-04T08:29:50.746 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:50.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:50.749 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:50.801 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:50.805 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:51.011 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.17.5","timestamp":"2020-06-04T08:29:31.959434Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:29:51.012 WARNING:rgw_multi.tests:bucket gpxthc-13 zone test-zone2 behind zone test-zone1: shard=1 master=00000000002.19.5 target=00000000001.17.5 2020-06-04T08:29:56.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:56.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:56.019 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:56.061 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:56.061 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:56.278 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.19.5","timestamp":"2020-06-04T08:29:31.759602Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:29:56.279 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-13 2020-06-04T08:29:56.280 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-13 zones={test-zone2, test-zone1} 2020-06-04T08:29:56.299 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:29:56.350 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:29:56.425 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-13 zones={test-zone2, test-zone1} 2020-06-04T08:29:56.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:56.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:56.428 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:56.478 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:56.479 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:56.682 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.19.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:29:56.682 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-13 2020-06-04T08:29:56.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:56.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:56.684 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:56.766 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:56.767 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:56.997 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.19.5","timestamp":"2020-06-04T08:29:31.759602Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:29:56.998 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-13 2020-06-04T08:29:56.999 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-13 zones={test-zone3, test-zone1} 2020-06-04T08:29:57.032 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:29:57.086 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:29:57.158 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-13 zones={test-zone3, test-zone1} 2020-06-04T08:29:57.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:57.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:57.160 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:57.201 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:57.202 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:57.430 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.19.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:29:57.430 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-13 2020-06-04T08:29:57.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:57.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:57.432 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:57.480 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:57.481 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:57.756 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.19.5","timestamp":"2020-06-04T08:29:31.759602Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:29:57.757 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-13 2020-06-04T08:29:57.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:57.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:57.786 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:57.845 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:57.846 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:58.048 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.16.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:29:58.049 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-14 2020-06-04T08:29:58.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:58.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:29:58.052 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:29:58.118 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:29:58.119 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:29:58.347 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.14.5","timestamp":"2020-06-04T08:29:31.941462Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:29:58.348 WARNING:rgw_multi.tests:bucket gpxthc-14 zone test-zone1 behind zone test-zone2: shard=1 master=00000000002.16.5 target=00000000001.14.5 2020-06-04T08:30:03.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:03.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:03.353 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:03.401 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:03.401 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:03.636 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.14.5","timestamp":"2020-06-04T08:29:31.941462Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:30:03.637 WARNING:rgw_multi.tests:bucket gpxthc-14 zone test-zone1 behind zone test-zone2: shard=1 master=00000000002.16.5 target=00000000001.14.5 2020-06-04T08:30:08.639 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:08.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:08.641 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:08.690 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:08.691 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:08.919 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.14.5","timestamp":"2020-06-04T08:29:31.941462Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:30:08.920 WARNING:rgw_multi.tests:bucket gpxthc-14 zone test-zone1 behind zone test-zone2: shard=1 master=00000000002.16.5 target=00000000001.14.5 2020-06-04T08:30:13.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:13.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:13.923 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:13.971 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:13.972 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:14.208 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.16.5","timestamp":"2020-06-04T08:29:31.812720Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:30:14.209 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-14 2020-06-04T08:30:14.211 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-14 zones={test-zone1, test-zone2} 2020-06-04T08:30:14.227 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:30:14.278 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:30:14.345 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-14 zones={test-zone1, test-zone2} 2020-06-04T08:30:14.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:14.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:14.347 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:14.385 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:14.385 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:14.590 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.16.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:30:14.590 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-14 2020-06-04T08:30:14.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:14.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:14.592 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:14.640 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:14.641 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:14.862 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.16.5","timestamp":"2020-06-04T08:29:31.812720Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:30:14.863 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-14 2020-06-04T08:30:14.863 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-14 zones={test-zone3, test-zone2} 2020-06-04T08:30:14.876 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:30:14.926 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:30:14.999 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-14 zones={test-zone3, test-zone2} 2020-06-04T08:30:15.032 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:15.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:15.034 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:15.074 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:15.075 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:15.280 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.13.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:30:15.281 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-15 2020-06-04T08:30:15.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:15.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:15.284 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:15.366 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:15.367 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:15.599 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.13.5","timestamp":"2020-06-04T08:29:31.866648Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:30:15.600 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-15 2020-06-04T08:30:15.600 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-15 zones={test-zone1, test-zone3} 2020-06-04T08:30:15.611 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:30:15.666 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:30:15.739 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-15 zones={test-zone1, test-zone3} 2020-06-04T08:30:15.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:15.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:15.741 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:15.786 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:15.786 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:16.001 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.13.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:30:16.001 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-15 2020-06-04T08:30:16.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:16.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:16.003 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-15 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:16.046 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:16.047 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:16.264 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.13.5","timestamp":"2020-06-04T08:29:31.866648Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:30:16.265 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-15 2020-06-04T08:30:16.266 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-15 zones={test-zone2, test-zone3} 2020-06-04T08:30:16.283 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:30:16.330 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:30:16.403 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-15 zones={test-zone2, test-zone3} 2020-06-04T08:30:16.406 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... ok 2020-06-04T08:30:16.408 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-16 2020-06-04T08:30:16.442 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gpxthc-17 2020-06-04T08:30:16.497 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gpxthc-18 2020-06-04T08:30:16.604 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:16.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:16.606 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:16.647 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:16.805 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=838ed6b2-fa2f-4917-a5c1-6c82bb405cda 2020-06-04T08:30:16.837 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259416.599911_110.1","last_update":"2020-06-04T08:30:16.599911Z"},{"marker":"1_1591259350.985454_111.1","last_update":"2020-06-04T08:29:10.985454Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:30:16.837 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:30:16.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:16.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:16.840 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:16.882 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:16.883 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:17.090 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259416.599911_110.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:30:16.599911Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:30:17.091 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:30:17.092 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:30:17.093 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:17.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:17.095 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:17.138 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:17.139 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:17.356 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259416.599911_110.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:30:16.599911Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:30:17.357 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:30:17.358 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:30:17.358 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:17.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:17.359 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:17.405 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:17.405 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:17.674 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259416.599911_110.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:30:16.599911Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:30:17.675 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:30:17.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:17.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:17.725 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:17.764 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:17.765 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:17.955 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.14.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:30:17.955 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-16 2020-06-04T08:30:17.956 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:17.956 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:17.957 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:18.001 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:18.001 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:18.207 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"2DSSx2QWbyOFmjRbgzRcuCkYWH8Co2y","ns":""},"count":1},"inc_marker":{"position":"00000000002.14.10","timestamp":"2020-06-04T08:30:17.779514Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:30:18.208 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-16 2020-06-04T08:30:18.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:18.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:18.211 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:18.255 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:18.255 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:18.478 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.20.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:30:18.479 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-16 2020-06-04T08:30:18.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:18.482 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:18.484 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:18.565 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:18.566 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:18.811 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"2DSSx2QWbyOFmjRbgzRcuCkYWH8Co2y","ns":""},"count":1},"inc_marker":{"position":"00000000002.14.10","timestamp":"2020-06-04T08:30:17.777489Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:30:18.812 WARNING:rgw_multi.tests:bucket gpxthc-16 zone test-zone3 behind zone test-zone1: shard=5 master=00000000005.20.9 target=00000000002.14.10 2020-06-04T08:30:23.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:23.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:23.815 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:23.860 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:23.860 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:24.096 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"2DSSx2QWbyOFmjRbgzRcuCkYWH8Co2y","ns":""},"count":1},"inc_marker":{"position":"00000000002.14.10","timestamp":"2020-06-04T08:30:17.777489Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:30:24.097 WARNING:rgw_multi.tests:bucket gpxthc-16 zone test-zone3 behind zone test-zone1: shard=5 master=00000000005.20.9 target=00000000002.14.10 2020-06-04T08:30:29.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:29.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:29.103 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:29.150 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:29.150 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:29.394 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"2DSSx2QWbyOFmjRbgzRcuCkYWH8Co2y","ns":""},"count":1},"inc_marker":{"position":"00000000002.14.10","timestamp":"2020-06-04T08:30:17.777489Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:30:29.395 WARNING:rgw_multi.tests:bucket gpxthc-16 zone test-zone3 behind zone test-zone1: shard=5 master=00000000005.20.9 target=00000000002.14.10 2020-06-04T08:30:34.396 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:34.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:34.398 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:34.447 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:34.448 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:34.685 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"2DSSx2QWbyOFmjRbgzRcuCkYWH8Co2y","ns":""},"count":1},"inc_marker":{"position":"00000000005.20.9","timestamp":"2020-06-04T08:30:17.683994Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:30:34.686 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-16 2020-06-04T08:30:34.687 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:34.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:34.688 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:34.733 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:34.734 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:34.942 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000006.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:30:34.943 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-16 2020-06-04T08:30:34.943 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:34.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:34.945 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:34.993 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:34.994 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:35.259 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.20.9","timestamp":"2020-06-04T08:30:17.683994Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:30:35.259 WARNING:rgw_multi.tests:bucket gpxthc-16 zone test-zone4 behind zone test-zone1: shard=5 master=00000000006.22.9 target=00000000005.20.9 2020-06-04T08:30:40.261 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:40.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:40.263 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:40.310 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:40.310 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:40.584 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.20.9","timestamp":"2020-06-04T08:30:17.683994Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:30:40.585 WARNING:rgw_multi.tests:bucket gpxthc-16 zone test-zone4 behind zone test-zone1: shard=5 master=00000000006.22.9 target=00000000005.20.9 2020-06-04T08:30:45.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:45.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:45.590 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:45.636 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:45.637 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:45.905 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.20.9","timestamp":"2020-06-04T08:30:17.683994Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:30:45.905 WARNING:rgw_multi.tests:bucket gpxthc-16 zone test-zone4 behind zone test-zone1: shard=5 master=00000000006.22.9 target=00000000005.20.9 2020-06-04T08:30:50.907 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:50.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:50.909 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:50.954 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:50.955 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:51.325 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.20.9","timestamp":"2020-06-04T08:30:17.683994Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:30:51.326 WARNING:rgw_multi.tests:bucket gpxthc-16 zone test-zone4 behind zone test-zone1: shard=5 master=00000000006.22.9 target=00000000005.20.9 2020-06-04T08:30:56.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:56.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:56.330 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:56.374 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:56.374 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:56.650 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.22.9","timestamp":"2020-06-04T08:30:17.683994Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:30:56.651 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-16 2020-06-04T08:30:56.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:56.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:56.704 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:56.744 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:56.745 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:56.940 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.14.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:30:56.940 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-17 2020-06-04T08:30:56.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:56.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:56.942 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:56.988 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:56.988 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:57.181 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000002.14.10","timestamp":"2020-06-04T08:30:56.789565Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:30:57.182 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-17 2020-06-04T08:30:57.183 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:57.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:57.184 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:57.226 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:57.227 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:57.432 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.18.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:30:57.432 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-17 2020-06-04T08:30:57.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:57.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:30:57.434 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:30:57.480 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:30:57.480 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:30:57.707 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000002.14.10","timestamp":"2020-06-04T08:30:56.789522Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:30:57.708 WARNING:rgw_multi.tests:bucket gpxthc-17 zone test-zone3 behind zone test-zone2: shard=5 master=00000000004.18.9 target=00000000002.14.10 2020-06-04T08:31:02.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:02.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:02.711 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:31:02.767 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:31:02.767 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:31:02.993 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000002.14.10","timestamp":"2020-06-04T08:30:56.789522Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:31:02.994 WARNING:rgw_multi.tests:bucket gpxthc-17 zone test-zone3 behind zone test-zone2: shard=5 master=00000000004.18.9 target=00000000002.14.10 2020-06-04T08:31:07.995 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:07.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:07.998 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:31:08.045 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:31:08.055 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:31:08.290 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000002.14.10","timestamp":"2020-06-04T08:30:56.789522Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:31:08.291 WARNING:rgw_multi.tests:bucket gpxthc-17 zone test-zone3 behind zone test-zone2: shard=5 master=00000000004.18.9 target=00000000002.14.10 2020-06-04T08:31:13.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:13.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:13.295 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:31:13.342 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:31:13.343 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:31:13.587 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000002.14.10","timestamp":"2020-06-04T08:30:56.789522Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:31:13.588 WARNING:rgw_multi.tests:bucket gpxthc-17 zone test-zone3 behind zone test-zone2: shard=5 master=00000000004.18.9 target=00000000002.14.10 2020-06-04T08:31:18.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:18.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:18.596 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:31:18.644 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:31:18.644 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:31:18.863 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.9","timestamp":"2020-06-04T08:30:56.665050Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:31:18.864 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-17 2020-06-04T08:31:18.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:18.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:18.933 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:31:18.976 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:31:18.977 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:31:19.186 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.23.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:31:19.187 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-18 2020-06-04T08:31:19.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:19.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:19.189 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:31:19.274 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:31:19.274 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:31:19.493 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"LThDOxUGE3zRpi0VYfsypl.WVvrhPWL","ns":""},"count":1},"inc_marker":{"position":"00000000002.23.10","timestamp":"2020-06-04T08:31:19.081413Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:31:19.494 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-18 2020-06-04T08:31:19.495 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:19.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:19.497 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:31:19.588 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:31:19.588 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:31:19.797 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000006.30.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:31:19.798 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-18 2020-06-04T08:31:19.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:19.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:19.801 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:31:19.858 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:31:19.859 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:31:20.064 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"LThDOxUGE3zRpi0VYfsypl.WVvrhPWL","ns":""},"count":1},"inc_marker":{"position":"00000000002.23.10","timestamp":"2020-06-04T08:31:19.083459Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:31:20.065 WARNING:rgw_multi.tests:bucket gpxthc-18 zone test-zone2 behind zone test-zone3: shard=5 master=00000000006.30.9 target=00000000002.23.10 2020-06-04T08:31:25.066 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:25.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:25.068 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:31:25.120 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:31:25.120 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:31:25.343 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"LThDOxUGE3zRpi0VYfsypl.WVvrhPWL","ns":""},"count":1},"inc_marker":{"position":"00000000002.23.10","timestamp":"2020-06-04T08:31:19.083459Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:31:25.344 WARNING:rgw_multi.tests:bucket gpxthc-18 zone test-zone2 behind zone test-zone3: shard=5 master=00000000006.30.9 target=00000000002.23.10 2020-06-04T08:31:30.347 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:30.348 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:30.349 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:31:30.395 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:31:30.396 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:31:30.628 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"LThDOxUGE3zRpi0VYfsypl.WVvrhPWL","ns":""},"count":1},"inc_marker":{"position":"00000000002.23.10","timestamp":"2020-06-04T08:31:19.083459Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:31:30.629 WARNING:rgw_multi.tests:bucket gpxthc-18 zone test-zone2 behind zone test-zone3: shard=5 master=00000000006.30.9 target=00000000002.23.10 2020-06-04T08:31:35.630 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:35.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:35.631 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:31:35.678 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:31:35.679 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:31:35.908 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"LThDOxUGE3zRpi0VYfsypl.WVvrhPWL","ns":""},"count":1},"inc_marker":{"position":"00000000002.23.10","timestamp":"2020-06-04T08:31:19.083459Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:31:35.909 WARNING:rgw_multi.tests:bucket gpxthc-18 zone test-zone2 behind zone test-zone3: shard=5 master=00000000006.30.9 target=00000000002.23.10 2020-06-04T08:31:40.910 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:40.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:40.912 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:31:40.964 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:31:40.964 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:31:41.200 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"LThDOxUGE3zRpi0VYfsypl.WVvrhPWL","ns":""},"count":1},"inc_marker":{"position":"00000000006.30.9","timestamp":"2020-06-04T08:31:18.884161Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:31:41.201 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-18 2020-06-04T08:31:43.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:43.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:43.059 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:31:43.108 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:31:43.109 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:31:43.299 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.27.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.20.9"},{"key":4,"val":""},{"key":5,"val":"00000000009.28.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:31:43.299 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-16 2020-06-04T08:31:43.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:43.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:43.301 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:31:43.348 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:31:43.349 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:31:43.590 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"NfY3L6fRpg1whtQiLmX2LV3cIdXDG4X","ns":""},"count":1},"inc_marker":{"position":"00000000010.27.5","timestamp":"2020-06-04T08:31:41.356153Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UkHrfSL8nXcwI9eHkyBaUGeH9CroVFY","ns":""},"count":1},"inc_marker":{"position":"00000000002.18.10","timestamp":"2020-06-04T08:31:41.819497Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"2DSSx2QWbyOFmjRbgzRcuCkYWH8Co2y","ns":""},"count":1},"inc_marker":{"position":"00000000009.28.9","timestamp":"2020-06-04T08:31:41.629203Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:31:43.591 WARNING:rgw_multi.tests:bucket gpxthc-16 zone test-zone2 behind zone test-zone1: shard=3 master=00000000003.20.9 target=00000000002.18.10 2020-06-04T08:31:48.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:48.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:48.596 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:31:48.642 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:31:48.643 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:31:48.876 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"NfY3L6fRpg1whtQiLmX2LV3cIdXDG4X","ns":""},"count":1},"inc_marker":{"position":"00000000010.27.5","timestamp":"2020-06-04T08:31:41.356153Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UkHrfSL8nXcwI9eHkyBaUGeH9CroVFY","ns":""},"count":1},"inc_marker":{"position":"00000000002.18.10","timestamp":"2020-06-04T08:31:41.819497Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"2DSSx2QWbyOFmjRbgzRcuCkYWH8Co2y","ns":""},"count":1},"inc_marker":{"position":"00000000009.28.9","timestamp":"2020-06-04T08:31:41.629203Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:31:48.877 WARNING:rgw_multi.tests:bucket gpxthc-16 zone test-zone2 behind zone test-zone1: shard=3 master=00000000003.20.9 target=00000000002.18.10 2020-06-04T08:31:53.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:53.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:53.883 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:31:53.937 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:31:53.938 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:31:54.152 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"NfY3L6fRpg1whtQiLmX2LV3cIdXDG4X","ns":""},"count":1},"inc_marker":{"position":"00000000010.27.5","timestamp":"2020-06-04T08:31:41.356153Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UkHrfSL8nXcwI9eHkyBaUGeH9CroVFY","ns":""},"count":1},"inc_marker":{"position":"00000000002.18.10","timestamp":"2020-06-04T08:31:41.819497Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"2DSSx2QWbyOFmjRbgzRcuCkYWH8Co2y","ns":""},"count":1},"inc_marker":{"position":"00000000009.28.9","timestamp":"2020-06-04T08:31:41.629203Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:31:54.153 WARNING:rgw_multi.tests:bucket gpxthc-16 zone test-zone2 behind zone test-zone1: shard=3 master=00000000003.20.9 target=00000000002.18.10 2020-06-04T08:31:59.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:59.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:31:59.156 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:31:59.199 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:31:59.200 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:31:59.419 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"NfY3L6fRpg1whtQiLmX2LV3cIdXDG4X","ns":""},"count":1},"inc_marker":{"position":"00000000010.27.5","timestamp":"2020-06-04T08:31:41.356153Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UkHrfSL8nXcwI9eHkyBaUGeH9CroVFY","ns":""},"count":1},"inc_marker":{"position":"00000000002.18.10","timestamp":"2020-06-04T08:31:41.819497Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"2DSSx2QWbyOFmjRbgzRcuCkYWH8Co2y","ns":""},"count":1},"inc_marker":{"position":"00000000009.28.9","timestamp":"2020-06-04T08:31:41.629203Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:31:59.419 WARNING:rgw_multi.tests:bucket gpxthc-16 zone test-zone2 behind zone test-zone1: shard=3 master=00000000003.20.9 target=00000000002.18.10 2020-06-04T08:32:04.426 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:04.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:04.428 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:04.487 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:04.487 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:04.716 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"NfY3L6fRpg1whtQiLmX2LV3cIdXDG4X","ns":""},"count":1},"inc_marker":{"position":"00000000010.27.5","timestamp":"2020-06-04T08:31:41.356153Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UkHrfSL8nXcwI9eHkyBaUGeH9CroVFY","ns":""},"count":1},"inc_marker":{"position":"00000000003.20.9","timestamp":"2020-06-04T08:31:41.438769Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"2DSSx2QWbyOFmjRbgzRcuCkYWH8Co2y","ns":""},"count":1},"inc_marker":{"position":"00000000009.28.9","timestamp":"2020-06-04T08:31:41.629203Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:32:04.717 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-16 2020-06-04T08:32:04.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:04.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:04.723 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:04.765 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:04.766 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:04.984 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.27.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.20.9"},{"key":4,"val":""},{"key":5,"val":"00000000009.28.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:32:04.985 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-16 2020-06-04T08:32:04.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:04.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:04.987 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:05.076 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:05.319 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"NfY3L6fRpg1whtQiLmX2LV3cIdXDG4X","ns":""},"count":1},"inc_marker":{"position":"00000000010.27.5","timestamp":"2020-06-04T08:31:41.356153Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UkHrfSL8nXcwI9eHkyBaUGeH9CroVFY","ns":""},"count":1},"inc_marker":{"position":"00000000003.20.9","timestamp":"2020-06-04T08:31:41.438769Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"2DSSx2QWbyOFmjRbgzRcuCkYWH8Co2y","ns":""},"count":1},"inc_marker":{"position":"00000000009.28.9","timestamp":"2020-06-04T08:31:41.629203Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:32:05.320 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-16 2020-06-04T08:32:05.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:05.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:05.322 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:05.402 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:05.402 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:05.624 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.27.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.20.9"},{"key":4,"val":""},{"key":5,"val":"00000000009.28.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:32:05.625 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-16 2020-06-04T08:32:05.625 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:05.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:05.627 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:05.687 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:05.688 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:05.948 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.27.5","timestamp":"2020-06-04T08:31:41.356153Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.20.9","timestamp":"2020-06-04T08:31:41.438769Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000009.28.9","timestamp":"2020-06-04T08:31:41.629203Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:32:05.949 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-16 2020-06-04T08:32:05.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:05.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:05.951 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:06.033 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:06.033 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:06.233 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.16.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.36.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.26.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:32:06.233 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-16 2020-06-04T08:32:06.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:06.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:06.236 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:06.288 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:06.289 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:06.539 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"NfY3L6fRpg1whtQiLmX2LV3cIdXDG4X","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.9","timestamp":"2020-06-04T08:31:41.229616Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UkHrfSL8nXcwI9eHkyBaUGeH9CroVFY","ns":""},"count":1},"inc_marker":{"position":"00000000010.36.5","timestamp":"2020-06-04T08:31:41.553400Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"2DSSx2QWbyOFmjRbgzRcuCkYWH8Co2y","ns":""},"count":1},"inc_marker":{"position":"00000000007.26.9","timestamp":"2020-06-04T08:31:41.629203Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:32:06.540 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-16 2020-06-04T08:32:06.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:06.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:06.555 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:06.596 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:06.597 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:06.799 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.16.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.36.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.26.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:32:06.800 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-16 2020-06-04T08:32:06.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:06.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:06.802 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:06.861 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:06.861 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:07.116 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"NfY3L6fRpg1whtQiLmX2LV3cIdXDG4X","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.9","timestamp":"2020-06-04T08:31:41.229616Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UkHrfSL8nXcwI9eHkyBaUGeH9CroVFY","ns":""},"count":1},"inc_marker":{"position":"00000000010.36.5","timestamp":"2020-06-04T08:31:41.553400Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"2DSSx2QWbyOFmjRbgzRcuCkYWH8Co2y","ns":""},"count":1},"inc_marker":{"position":"00000000007.26.9","timestamp":"2020-06-04T08:31:41.629203Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:32:07.117 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-16 2020-06-04T08:32:07.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:07.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:07.122 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:07.201 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:07.201 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:07.421 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.18.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.20.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.36.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:32:07.422 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-16 2020-06-04T08:32:07.422 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:07.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:07.424 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:07.514 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:07.515 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:07.738 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"NfY3L6fRpg1whtQiLmX2LV3cIdXDG4X","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.9","timestamp":"2020-06-04T08:31:41.229616Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UkHrfSL8nXcwI9eHkyBaUGeH9CroVFY","ns":""},"count":1},"inc_marker":{"position":"00000000003.20.9","timestamp":"2020-06-04T08:31:41.438769Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"2DSSx2QWbyOFmjRbgzRcuCkYWH8Co2y","ns":""},"count":1},"inc_marker":{"position":"00000000014.36.5","timestamp":"2020-06-04T08:31:41.757106Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:32:07.738 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-16 2020-06-04T08:32:07.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:07.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:07.740 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:07.820 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:07.821 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:08.042 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.18.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.20.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.36.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:32:08.042 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-16 2020-06-04T08:32:08.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:08.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:08.044 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:08.083 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:08.084 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:08.317 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"NfY3L6fRpg1whtQiLmX2LV3cIdXDG4X","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.9","timestamp":"2020-06-04T08:31:41.229616Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UkHrfSL8nXcwI9eHkyBaUGeH9CroVFY","ns":""},"count":1},"inc_marker":{"position":"00000000003.20.9","timestamp":"2020-06-04T08:31:41.438769Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"2DSSx2QWbyOFmjRbgzRcuCkYWH8Co2y","ns":""},"count":1},"inc_marker":{"position":"00000000014.36.5","timestamp":"2020-06-04T08:31:41.757106Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:32:08.318 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-16 2020-06-04T08:32:08.319 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-16 zones={test-zone2, test-zone1} 2020-06-04T08:32:08.345 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:32:08.354 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:32:08.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:32:08.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:32:08.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:32:08.358 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:32:08.373 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:32:08.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:32:08.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:32:08.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:32:08.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:32:08.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:32:08.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:32:08.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:32:08.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:32:08.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:32:08.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:32:08.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:32:09.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:32:09.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:32:09.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:32:09.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:32:09.513 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-16 zones={test-zone2, test-zone1} 2020-06-04T08:32:09.514 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-16 zones={test-zone3, test-zone1} 2020-06-04T08:32:09.523 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:32:09.531 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:32:09.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:32:09.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:32:09.533 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:32:09.533 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:32:09.541 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:32:09.542 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:32:09.543 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:32:09.544 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:32:09.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:32:09.578 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:32:09.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:32:09.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:32:09.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:32:09.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:32:10.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:32:10.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:32:10.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:32:10.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:32:10.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:32:10.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:32:10.617 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-16 zones={test-zone3, test-zone1} 2020-06-04T08:32:10.618 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-16 zones={test-zone3, test-zone2} 2020-06-04T08:32:10.644 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:32:10.655 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:32:10.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:32:10.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:32:10.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:32:10.658 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:32:10.667 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:32:10.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:32:10.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:32:10.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:32:10.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:32:10.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:32:10.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:32:10.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:32:10.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:32:11.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:32:11.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:32:11.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:32:11.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:32:11.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:32:11.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:32:11.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:32:11.759 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-16 zones={test-zone3, test-zone2} 2020-06-04T08:32:11.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:11.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:11.761 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:11.805 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:11.806 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:12.017 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.34.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.34.9"},{"key":4,"val":""},{"key":5,"val":"00000000006.18.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:32:12.017 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-17 2020-06-04T08:32:12.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:12.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:12.019 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:12.064 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:12.064 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:12.291 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"Qh.6cmS.CaDDDs-.d7LiIqDavhXLq3s","ns":""},"count":1},"inc_marker":{"position":"00000000010.34.5","timestamp":"2020-06-04T08:31:41.960714Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"KFtGJlLDp2kIROKHyRpH9dymw8TQG.B","ns":""},"count":1},"inc_marker":{"position":"00000000003.34.9","timestamp":"2020-06-04T08:31:42.045618Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000004.14.9","timestamp":"2020-06-04T08:30:56.665050Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:32:12.291 WARNING:rgw_multi.tests:bucket gpxthc-17 zone test-zone2 behind zone test-zone1: shard=5 master=00000000006.18.10 target=00000000004.14.9 2020-06-04T08:32:17.293 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:17.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:17.296 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:17.350 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:17.351 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:17.578 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"Qh.6cmS.CaDDDs-.d7LiIqDavhXLq3s","ns":""},"count":1},"inc_marker":{"position":"00000000010.34.5","timestamp":"2020-06-04T08:31:41.960714Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"KFtGJlLDp2kIROKHyRpH9dymw8TQG.B","ns":""},"count":1},"inc_marker":{"position":"00000000003.34.9","timestamp":"2020-06-04T08:31:42.045618Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000004.14.9","timestamp":"2020-06-04T08:30:56.665050Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:32:17.579 WARNING:rgw_multi.tests:bucket gpxthc-17 zone test-zone2 behind zone test-zone1: shard=5 master=00000000006.18.10 target=00000000004.14.9 2020-06-04T08:32:22.582 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:22.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:22.584 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:22.626 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:22.626 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:22.843 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"Qh.6cmS.CaDDDs-.d7LiIqDavhXLq3s","ns":""},"count":1},"inc_marker":{"position":"00000000010.34.5","timestamp":"2020-06-04T08:31:41.960714Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"KFtGJlLDp2kIROKHyRpH9dymw8TQG.B","ns":""},"count":1},"inc_marker":{"position":"00000000003.34.9","timestamp":"2020-06-04T08:31:42.045618Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000006.18.10","timestamp":"2020-06-04T08:31:42.243131Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:32:22.844 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-17 2020-06-04T08:32:22.844 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:22.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:22.846 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:22.888 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:22.889 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:23.102 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.34.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.34.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.20.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:32:23.103 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-17 2020-06-04T08:32:23.103 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:23.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:23.105 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:23.189 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:23.189 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:23.428 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"Qh.6cmS.CaDDDs-.d7LiIqDavhXLq3s","ns":""},"count":1},"inc_marker":{"position":"00000000010.34.5","timestamp":"2020-06-04T08:31:41.960714Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"KFtGJlLDp2kIROKHyRpH9dymw8TQG.B","ns":""},"count":1},"inc_marker":{"position":"00000000003.34.9","timestamp":"2020-06-04T08:31:42.045618Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000007.20.9","timestamp":"2020-06-04T08:31:42.243131Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:32:23.429 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-17 2020-06-04T08:32:23.430 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:23.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:23.434 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:23.479 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:23.479 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:23.694 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.34.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.34.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.20.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:32:23.695 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-17 2020-06-04T08:32:23.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:23.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:23.697 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:23.742 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:23.742 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:24.008 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.34.5","timestamp":"2020-06-04T08:31:41.960714Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.34.9","timestamp":"2020-06-04T08:31:42.045618Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.18.10","timestamp":"2020-06-04T08:31:42.243131Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:32:24.009 WARNING:rgw_multi.tests:bucket gpxthc-17 zone test-zone4 behind zone test-zone1: shard=5 master=00000000007.20.9 target=00000000006.18.10 2020-06-04T08:32:29.010 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:29.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:29.012 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:29.059 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:29.059 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:29.325 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.34.5","timestamp":"2020-06-04T08:31:41.960714Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.34.9","timestamp":"2020-06-04T08:31:42.045618Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.18.10","timestamp":"2020-06-04T08:31:42.243131Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:32:29.326 WARNING:rgw_multi.tests:bucket gpxthc-17 zone test-zone4 behind zone test-zone1: shard=5 master=00000000007.20.9 target=00000000006.18.10 2020-06-04T08:32:34.328 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:34.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:34.330 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:34.376 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:34.377 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:34.644 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.34.5","timestamp":"2020-06-04T08:31:41.960714Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.34.9","timestamp":"2020-06-04T08:31:42.045618Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.18.10","timestamp":"2020-06-04T08:31:42.243131Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:32:34.645 WARNING:rgw_multi.tests:bucket gpxthc-17 zone test-zone4 behind zone test-zone1: shard=5 master=00000000007.20.9 target=00000000006.18.10 2020-06-04T08:32:39.647 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:39.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:39.648 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:39.693 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:39.693 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:39.966 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.34.5","timestamp":"2020-06-04T08:31:41.960714Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.34.9","timestamp":"2020-06-04T08:31:42.045618Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.18.10","timestamp":"2020-06-04T08:31:42.243131Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:32:39.966 WARNING:rgw_multi.tests:bucket gpxthc-17 zone test-zone4 behind zone test-zone1: shard=5 master=00000000007.20.9 target=00000000006.18.10 2020-06-04T08:32:44.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:44.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:44.972 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:45.017 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:45.018 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:45.284 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.34.5","timestamp":"2020-06-04T08:31:41.960714Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.34.9","timestamp":"2020-06-04T08:31:42.045618Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.18.10","timestamp":"2020-06-04T08:31:42.243131Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:32:45.285 WARNING:rgw_multi.tests:bucket gpxthc-17 zone test-zone4 behind zone test-zone1: shard=5 master=00000000007.20.9 target=00000000006.18.10 2020-06-04T08:32:50.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:50.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:50.288 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:50.334 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:50.334 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:50.601 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.34.5","timestamp":"2020-06-04T08:31:41.960714Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.34.9","timestamp":"2020-06-04T08:31:42.045618Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.18.10","timestamp":"2020-06-04T08:31:42.243131Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:32:50.602 WARNING:rgw_multi.tests:bucket gpxthc-17 zone test-zone4 behind zone test-zone1: shard=5 master=00000000007.20.9 target=00000000006.18.10 2020-06-04T08:32:55.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:55.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:55.614 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:55.658 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:55.658 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:55.927 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.34.5","timestamp":"2020-06-04T08:31:41.960714Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.34.9","timestamp":"2020-06-04T08:31:42.045618Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.20.9","timestamp":"2020-06-04T08:31:42.243131Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:32:55.928 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-17 2020-06-04T08:32:55.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:55.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:55.930 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:56.010 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:56.010 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:56.201 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.18.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.46.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.26.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:32:56.202 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-17 2020-06-04T08:32:56.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:56.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:56.204 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:56.260 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:56.261 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:56.493 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"Qh.6cmS.CaDDDs-.d7LiIqDavhXLq3s","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.9","timestamp":"2020-06-04T08:31:41.841731Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"KFtGJlLDp2kIROKHyRpH9dymw8TQG.B","ns":""},"count":1},"inc_marker":{"position":"00000000010.46.5","timestamp":"2020-06-04T08:31:42.162876Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000008.26.9","timestamp":"2020-06-04T08:31:42.243131Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:32:56.494 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-17 2020-06-04T08:32:56.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:56.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:56.496 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:56.541 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:56.541 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:56.751 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.18.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.46.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.26.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:32:56.752 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-17 2020-06-04T08:32:56.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:56.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:56.753 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:56.798 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:56.799 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:57.035 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"Qh.6cmS.CaDDDs-.d7LiIqDavhXLq3s","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.9","timestamp":"2020-06-04T08:31:41.841731Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"KFtGJlLDp2kIROKHyRpH9dymw8TQG.B","ns":""},"count":1},"inc_marker":{"position":"00000000010.46.5","timestamp":"2020-06-04T08:31:42.162876Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000008.26.9","timestamp":"2020-06-04T08:31:42.243131Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:32:57.036 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-17 2020-06-04T08:32:57.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:57.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:57.038 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:57.119 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:57.120 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:57.330 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.17.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.42.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.38.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:32:57.331 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-17 2020-06-04T08:32:57.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:57.332 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:57.333 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:57.418 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:57.418 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:57.656 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"Qh.6cmS.CaDDDs-.d7LiIqDavhXLq3s","ns":""},"count":1},"inc_marker":{"position":"00000000003.17.9","timestamp":"2020-06-04T08:31:41.841731Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"KFtGJlLDp2kIROKHyRpH9dymw8TQG.B","ns":""},"count":1},"inc_marker":{"position":"00000000003.42.9","timestamp":"2020-06-04T08:31:42.045618Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000014.38.5","timestamp":"2020-06-04T08:31:42.363900Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:32:57.657 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-17 2020-06-04T08:32:57.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:57.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:57.660 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:57.738 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:57.739 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:57.967 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.17.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.42.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.38.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:32:57.968 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-17 2020-06-04T08:32:57.968 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:57.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:32:57.969 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:32:58.014 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:32:58.015 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:32:58.236 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"Qh.6cmS.CaDDDs-.d7LiIqDavhXLq3s","ns":""},"count":1},"inc_marker":{"position":"00000000003.17.9","timestamp":"2020-06-04T08:31:41.841731Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"KFtGJlLDp2kIROKHyRpH9dymw8TQG.B","ns":""},"count":1},"inc_marker":{"position":"00000000003.42.9","timestamp":"2020-06-04T08:31:42.045618Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000014.38.5","timestamp":"2020-06-04T08:31:42.363900Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:32:58.237 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-17 2020-06-04T08:32:58.238 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-17 zones={test-zone2, test-zone1} 2020-06-04T08:32:58.251 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:32:58.260 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:32:58.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:32:58.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:32:58.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:32:58.262 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:32:58.273 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:32:58.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:32:58.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:32:58.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:32:58.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:32:58.302 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:32:58.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:32:58.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:32:58.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:32:58.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:32:58.748 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:32:58.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:32:58.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:32:59.040 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:32:59.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:32:59.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:32:59.333 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-17 zones={test-zone2, test-zone1} 2020-06-04T08:32:59.334 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-17 zones={test-zone3, test-zone1} 2020-06-04T08:32:59.347 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:32:59.354 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:32:59.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:32:59.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:32:59.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:32:59.356 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:32:59.363 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:32:59.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:32:59.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:32:59.365 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:32:59.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:32:59.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:32:59.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:32:59.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:32:59.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:32:59.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:32:59.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:32:59.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:00.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:00.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:00.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:00.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:00.428 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-17 zones={test-zone3, test-zone1} 2020-06-04T08:33:00.429 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-17 zones={test-zone3, test-zone2} 2020-06-04T08:33:00.440 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:33:00.450 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:00.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:00.451 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:00.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:00.452 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:33:00.459 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:00.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:00.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:00.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:00.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:00.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:00.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:00.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:00.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:00.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:00.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:00.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:01.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:01.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:01.294 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:01.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:01.535 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-17 zones={test-zone3, test-zone2} 2020-06-04T08:33:01.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:01.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:01.536 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:01.581 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:01.582 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:01.782 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.39.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.43.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.31.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:01.783 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-18 2020-06-04T08:33:01.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:01.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:01.785 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:01.832 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:01.833 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:02.066 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a3aWorHM27xLX.kZUeX8g5LZc2lJmxR","ns":""},"count":1},"inc_marker":{"position":"00000000010.39.5","timestamp":"2020-06-04T08:31:42.574096Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"n8qICeGtZKx3J-ImEl0czYpahQFAdgv","ns":""},"count":1},"inc_marker":{"position":"00000000003.43.9","timestamp":"2020-06-04T08:31:42.653188Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"LThDOxUGE3zRpi0VYfsypl.WVvrhPWL","ns":""},"count":1},"inc_marker":{"position":"00000000007.31.9","timestamp":"2020-06-04T08:31:42.874057Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:02.067 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-18 2020-06-04T08:33:02.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:02.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:02.068 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:02.112 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:02.112 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:02.337 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.39.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.43.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.31.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:02.337 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-18 2020-06-04T08:33:02.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:02.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:02.339 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:02.425 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:02.425 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:02.646 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a3aWorHM27xLX.kZUeX8g5LZc2lJmxR","ns":""},"count":1},"inc_marker":{"position":"00000000010.39.5","timestamp":"2020-06-04T08:31:42.574096Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"n8qICeGtZKx3J-ImEl0czYpahQFAdgv","ns":""},"count":1},"inc_marker":{"position":"00000000003.43.9","timestamp":"2020-06-04T08:31:42.653188Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"LThDOxUGE3zRpi0VYfsypl.WVvrhPWL","ns":""},"count":1},"inc_marker":{"position":"00000000007.31.9","timestamp":"2020-06-04T08:31:42.874057Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:02.647 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-18 2020-06-04T08:33:02.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:02.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:02.649 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:02.689 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:02.690 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:02.911 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.39.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.43.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.31.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:02.912 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-18 2020-06-04T08:33:02.913 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:02.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:02.914 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:02.961 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:02.961 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:03.230 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.39.5","timestamp":"2020-06-04T08:31:42.574096Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.43.9","timestamp":"2020-06-04T08:31:42.653188Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.31.9","timestamp":"2020-06-04T08:31:42.874057Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:03.231 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-18 2020-06-04T08:33:03.231 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:03.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:03.232 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:03.274 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:03.275 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:03.494 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.23.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.38.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.30.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:03.495 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-18 2020-06-04T08:33:03.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:03.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:03.497 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:03.550 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:03.556 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:03.782 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a3aWorHM27xLX.kZUeX8g5LZc2lJmxR","ns":""},"count":1},"inc_marker":{"position":"00000000003.23.9","timestamp":"2020-06-04T08:31:42.451591Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"n8qICeGtZKx3J-ImEl0czYpahQFAdgv","ns":""},"count":2},"inc_marker":{"position":"00000000010.38.5","timestamp":"2020-06-04T08:31:42.794282Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"LThDOxUGE3zRpi0VYfsypl.WVvrhPWL","ns":""},"count":1},"inc_marker":{"position":"00000000007.30.9","timestamp":"2020-06-04T08:31:42.874057Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:03.783 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-18 2020-06-04T08:33:03.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:03.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:03.786 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:03.828 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:03.829 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:04.038 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.23.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.38.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.30.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:04.039 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-18 2020-06-04T08:33:04.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:04.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:04.041 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:04.088 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:04.088 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:04.336 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a3aWorHM27xLX.kZUeX8g5LZc2lJmxR","ns":""},"count":1},"inc_marker":{"position":"00000000003.23.9","timestamp":"2020-06-04T08:31:42.451591Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"n8qICeGtZKx3J-ImEl0czYpahQFAdgv","ns":""},"count":2},"inc_marker":{"position":"00000000010.38.5","timestamp":"2020-06-04T08:31:42.794282Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"LThDOxUGE3zRpi0VYfsypl.WVvrhPWL","ns":""},"count":1},"inc_marker":{"position":"00000000007.30.9","timestamp":"2020-06-04T08:31:42.874057Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:04.337 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-18 2020-06-04T08:33:04.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:04.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:04.340 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:04.390 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:04.390 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:04.608 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.22.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.23.9"},{"key":4,"val":""},{"key":5,"val":"00000000016.51.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:04.609 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-18 2020-06-04T08:33:04.609 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:04.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:04.610 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:04.695 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:04.695 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:04.940 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a3aWorHM27xLX.kZUeX8g5LZc2lJmxR","ns":""},"count":1},"inc_marker":{"position":"00000000003.22.9","timestamp":"2020-06-04T08:31:42.451591Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"n8qICeGtZKx3J-ImEl0czYpahQFAdgv","ns":""},"count":1},"inc_marker":{"position":"00000000003.23.9","timestamp":"2020-06-04T08:31:42.653188Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"LThDOxUGE3zRpi0VYfsypl.WVvrhPWL","ns":""},"count":1},"inc_marker":{"position":"00000000016.51.5","timestamp":"2020-06-04T08:31:42.994305Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:04.940 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-18 2020-06-04T08:33:04.941 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:04.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:04.942 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:05.023 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:05.024 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:05.247 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.22.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.23.9"},{"key":4,"val":""},{"key":5,"val":"00000000016.51.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:05.247 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-18 2020-06-04T08:33:05.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:05.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:05.249 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:05.295 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:05.296 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:05.532 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a3aWorHM27xLX.kZUeX8g5LZc2lJmxR","ns":""},"count":1},"inc_marker":{"position":"00000000003.22.9","timestamp":"2020-06-04T08:31:42.451591Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"n8qICeGtZKx3J-ImEl0czYpahQFAdgv","ns":""},"count":1},"inc_marker":{"position":"00000000003.23.9","timestamp":"2020-06-04T08:31:42.653188Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"LThDOxUGE3zRpi0VYfsypl.WVvrhPWL","ns":""},"count":1},"inc_marker":{"position":"00000000016.51.5","timestamp":"2020-06-04T08:31:42.994305Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:05.533 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-18 2020-06-04T08:33:05.533 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-18 zones={test-zone2, test-zone1} 2020-06-04T08:33:05.547 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:33:05.559 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:05.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:05.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:05.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:05.561 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:33:05.570 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:05.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:05.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:05.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:05.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:05.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:05.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:05.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:05.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:05.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:06.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:06.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:06.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:06.392 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:06.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:06.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:06.713 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-18 zones={test-zone2, test-zone1} 2020-06-04T08:33:06.713 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-18 zones={test-zone3, test-zone1} 2020-06-04T08:33:06.736 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:33:06.755 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:06.756 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:06.756 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:06.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:06.767 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:33:06.786 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:06.789 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:06.795 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:06.795 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:06.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:06.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:06.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:07.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:07.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:07.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:07.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:07.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:07.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:07.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:07.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:07.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:07.982 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-18 zones={test-zone3, test-zone1} 2020-06-04T08:33:07.983 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-18 zones={test-zone3, test-zone2} 2020-06-04T08:33:07.993 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:33:08.002 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:08.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:08.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:08.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:08.005 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:33:08.014 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:08.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:08.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:08.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:08.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:08.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:08.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:08.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:08.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:08.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:08.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:08.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:08.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:08.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:08.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:08.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:09.092 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-18 zones={test-zone3, test-zone2} 2020-06-04T08:33:09.768 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:09.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:09.770 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:09.808 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:09.809 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:10.010 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.34.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.23.5"},{"key":4,"val":""},{"key":5,"val":"00000000010.37.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:10.011 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-16 2020-06-04T08:33:10.011 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:10.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:10.013 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:10.055 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:10.055 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:10.264 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"NfY3L6fRpg1whtQiLmX2LV3cIdXDG4X","ns":""},"count":1},"inc_marker":{"position":"00000000011.34.5","timestamp":"2020-06-04T08:33:09.161400Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UkHrfSL8nXcwI9eHkyBaUGeH9CroVFY","ns":""},"count":1},"inc_marker":{"position":"00000000004.23.5","timestamp":"2020-06-04T08:33:09.237088Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"2DSSx2QWbyOFmjRbgzRcuCkYWH8Co2y","ns":""},"count":1},"inc_marker":{"position":"00000000010.37.5","timestamp":"2020-06-04T08:33:09.316986Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:10.265 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-16 2020-06-04T08:33:10.265 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:10.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:10.267 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:10.310 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:10.311 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:10.540 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.34.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.23.5"},{"key":4,"val":""},{"key":5,"val":"00000000010.37.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:10.541 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-16 2020-06-04T08:33:10.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:10.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:10.542 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:10.635 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:10.636 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:10.864 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"NfY3L6fRpg1whtQiLmX2LV3cIdXDG4X","ns":""},"count":1},"inc_marker":{"position":"00000000011.34.5","timestamp":"2020-06-04T08:33:09.161400Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UkHrfSL8nXcwI9eHkyBaUGeH9CroVFY","ns":""},"count":1},"inc_marker":{"position":"00000000004.23.5","timestamp":"2020-06-04T08:33:09.237088Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"2DSSx2QWbyOFmjRbgzRcuCkYWH8Co2y","ns":""},"count":1},"inc_marker":{"position":"00000000010.37.5","timestamp":"2020-06-04T08:33:09.316986Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:10.865 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-16 2020-06-04T08:33:10.866 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:10.866 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:10.867 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:10.919 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:10.920 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:11.142 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.34.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.23.5"},{"key":4,"val":""},{"key":5,"val":"00000000010.37.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:11.142 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-16 2020-06-04T08:33:11.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:11.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:11.144 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:11.185 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:11.185 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:11.455 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.34.5","timestamp":"2020-06-04T08:33:09.161400Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.23.5","timestamp":"2020-06-04T08:33:09.237088Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.37.5","timestamp":"2020-06-04T08:33:09.316986Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:11.456 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-16 2020-06-04T08:33:11.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:11.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:11.458 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:11.539 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:11.539 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:11.741 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.19.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.43.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.55.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:11.742 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-16 2020-06-04T08:33:11.742 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:11.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:11.743 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:11.794 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:11.794 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:12.035 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"NfY3L6fRpg1whtQiLmX2LV3cIdXDG4X","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.5","timestamp":"2020-06-04T08:33:09.161400Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UkHrfSL8nXcwI9eHkyBaUGeH9CroVFY","ns":""},"count":1},"inc_marker":{"position":"00000000011.43.5","timestamp":"2020-06-04T08:33:09.237088Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"2DSSx2QWbyOFmjRbgzRcuCkYWH8Co2y","ns":""},"count":1},"inc_marker":{"position":"00000000008.55.5","timestamp":"2020-06-04T08:33:09.316986Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:12.036 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-16 2020-06-04T08:33:12.037 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:12.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:12.038 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:12.081 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:12.082 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:12.290 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.19.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.43.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.55.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:12.291 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-16 2020-06-04T08:33:12.291 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:12.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:12.293 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:12.341 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:12.342 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:12.583 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"NfY3L6fRpg1whtQiLmX2LV3cIdXDG4X","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.5","timestamp":"2020-06-04T08:33:09.161400Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UkHrfSL8nXcwI9eHkyBaUGeH9CroVFY","ns":""},"count":1},"inc_marker":{"position":"00000000011.43.5","timestamp":"2020-06-04T08:33:09.237088Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"2DSSx2QWbyOFmjRbgzRcuCkYWH8Co2y","ns":""},"count":1},"inc_marker":{"position":"00000000008.55.5","timestamp":"2020-06-04T08:33:09.316986Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:12.584 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-16 2020-06-04T08:33:12.585 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:12.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:12.586 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:12.668 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:12.669 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:12.876 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.21.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.23.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.45.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:12.877 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-16 2020-06-04T08:33:12.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:12.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:12.879 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:12.962 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:12.962 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:13.217 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"NfY3L6fRpg1whtQiLmX2LV3cIdXDG4X","ns":""},"count":1},"inc_marker":{"position":"00000000004.21.5","timestamp":"2020-06-04T08:33:09.161400Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UkHrfSL8nXcwI9eHkyBaUGeH9CroVFY","ns":""},"count":1},"inc_marker":{"position":"00000000004.23.5","timestamp":"2020-06-04T08:33:09.237088Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"2DSSx2QWbyOFmjRbgzRcuCkYWH8Co2y","ns":""},"count":1},"inc_marker":{"position":"00000000015.45.5","timestamp":"2020-06-04T08:33:09.316986Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:13.218 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-16 2020-06-04T08:33:13.218 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:13.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:13.219 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:13.302 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:13.303 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:13.523 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.21.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.23.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.45.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:13.523 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-16 2020-06-04T08:33:13.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:13.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:13.525 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:13.569 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:13.569 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:13.793 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"NfY3L6fRpg1whtQiLmX2LV3cIdXDG4X","ns":""},"count":1},"inc_marker":{"position":"00000000004.21.5","timestamp":"2020-06-04T08:33:09.161400Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"UkHrfSL8nXcwI9eHkyBaUGeH9CroVFY","ns":""},"count":1},"inc_marker":{"position":"00000000004.23.5","timestamp":"2020-06-04T08:33:09.237088Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"2DSSx2QWbyOFmjRbgzRcuCkYWH8Co2y","ns":""},"count":1},"inc_marker":{"position":"00000000015.45.5","timestamp":"2020-06-04T08:33:09.316986Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:13.793 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-16 2020-06-04T08:33:13.794 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-16 zones={test-zone2, test-zone1} 2020-06-04T08:33:13.805 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:33:13.812 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:13.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:13.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:13.813 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:13.814 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:33:13.821 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:13.821 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:13.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:13.822 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:13.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:13.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:13.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:13.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:14.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:14.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:14.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:14.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:14.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:14.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:14.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:14.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:14.852 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-16 zones={test-zone2, test-zone1} 2020-06-04T08:33:14.853 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-16 zones={test-zone3, test-zone1} 2020-06-04T08:33:14.866 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:33:14.873 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:14.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:14.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:14.875 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:14.875 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:33:14.882 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:14.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:14.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:14.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:14.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:14.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:14.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:15.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:15.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:15.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:15.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:15.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:15.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:15.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:15.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:15.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:15.960 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-16 zones={test-zone3, test-zone1} 2020-06-04T08:33:15.961 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-16 zones={test-zone3, test-zone2} 2020-06-04T08:33:15.972 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:33:15.980 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:15.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:15.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:15.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:15.983 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:33:15.994 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:15.994 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:15.995 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:15.995 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:16.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:16.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:16.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:16.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:16.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:16.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:16.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:16.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:16.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:16.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:16.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:16.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:17.034 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-16 zones={test-zone3, test-zone2} 2020-06-04T08:33:17.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:17.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:17.036 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:17.077 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:17.078 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:17.296 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.47.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.39.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.23.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:17.297 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-17 2020-06-04T08:33:17.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:17.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:17.302 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:17.346 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:17.347 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:17.580 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"Qh.6cmS.CaDDDs-.d7LiIqDavhXLq3s","ns":""},"count":1},"inc_marker":{"position":"00000000011.47.5","timestamp":"2020-06-04T08:33:09.395305Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"KFtGJlLDp2kIROKHyRpH9dymw8TQG.B","ns":""},"count":1},"inc_marker":{"position":"00000000004.39.5","timestamp":"2020-06-04T08:33:09.476621Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000007.20.9","timestamp":"2020-06-04T08:31:42.243131Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:17.581 WARNING:rgw_multi.tests:bucket gpxthc-17 zone test-zone2 behind zone test-zone1: shard=5 master=00000000008.23.5 target=00000000007.20.9 2020-06-04T08:33:22.586 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:22.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:22.588 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:22.634 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:22.634 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:22.860 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"Qh.6cmS.CaDDDs-.d7LiIqDavhXLq3s","ns":""},"count":1},"inc_marker":{"position":"00000000011.47.5","timestamp":"2020-06-04T08:33:09.395305Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"KFtGJlLDp2kIROKHyRpH9dymw8TQG.B","ns":""},"count":1},"inc_marker":{"position":"00000000004.39.5","timestamp":"2020-06-04T08:33:09.476621Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000007.20.9","timestamp":"2020-06-04T08:31:42.243131Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:22.861 WARNING:rgw_multi.tests:bucket gpxthc-17 zone test-zone2 behind zone test-zone1: shard=5 master=00000000008.23.5 target=00000000007.20.9 2020-06-04T08:33:27.863 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:27.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:27.920 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:27.967 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:27.968 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:28.181 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"Qh.6cmS.CaDDDs-.d7LiIqDavhXLq3s","ns":""},"count":1},"inc_marker":{"position":"00000000011.47.5","timestamp":"2020-06-04T08:33:09.395305Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"KFtGJlLDp2kIROKHyRpH9dymw8TQG.B","ns":""},"count":1},"inc_marker":{"position":"00000000004.39.5","timestamp":"2020-06-04T08:33:09.476621Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000007.20.9","timestamp":"2020-06-04T08:31:42.243131Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:28.181 WARNING:rgw_multi.tests:bucket gpxthc-17 zone test-zone2 behind zone test-zone1: shard=5 master=00000000008.23.5 target=00000000007.20.9 2020-06-04T08:33:33.183 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:33.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:33.186 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:33.231 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:33.232 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:33.463 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"Qh.6cmS.CaDDDs-.d7LiIqDavhXLq3s","ns":""},"count":1},"inc_marker":{"position":"00000000011.47.5","timestamp":"2020-06-04T08:33:09.395305Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"KFtGJlLDp2kIROKHyRpH9dymw8TQG.B","ns":""},"count":1},"inc_marker":{"position":"00000000004.39.5","timestamp":"2020-06-04T08:33:09.476621Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000008.23.5","timestamp":"2020-06-04T08:33:09.550993Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:33.465 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-17 2020-06-04T08:33:33.466 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:33.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:33.468 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:33.529 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:33.529 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:33.750 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.47.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.39.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.23.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:33.751 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-17 2020-06-04T08:33:33.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:33.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:33.753 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:33.836 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:33.837 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:34.078 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"Qh.6cmS.CaDDDs-.d7LiIqDavhXLq3s","ns":""},"count":1},"inc_marker":{"position":"00000000011.47.5","timestamp":"2020-06-04T08:33:09.395305Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"KFtGJlLDp2kIROKHyRpH9dymw8TQG.B","ns":""},"count":1},"inc_marker":{"position":"00000000004.39.5","timestamp":"2020-06-04T08:33:09.476621Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000008.23.5","timestamp":"2020-06-04T08:33:09.550993Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:34.079 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-17 2020-06-04T08:33:34.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:34.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:34.081 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:34.123 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:34.123 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:34.356 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.47.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.39.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.23.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:34.357 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-17 2020-06-04T08:33:34.358 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:34.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:34.359 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:34.407 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:34.407 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:34.681 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.47.5","timestamp":"2020-06-04T08:33:09.395305Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.5","timestamp":"2020-06-04T08:33:09.476621Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.23.5","timestamp":"2020-06-04T08:33:09.550993Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:35.056 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-17 2020-06-04T08:33:35.057 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:35.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:35.059 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:35.101 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:35.101 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:35.305 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.45.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.53.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.29.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:35.306 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-17 2020-06-04T08:33:35.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:35.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:35.308 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:35.355 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:35.355 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:35.607 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"Qh.6cmS.CaDDDs-.d7LiIqDavhXLq3s","ns":""},"count":1},"inc_marker":{"position":"00000000004.45.5","timestamp":"2020-06-04T08:33:09.395305Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"KFtGJlLDp2kIROKHyRpH9dymw8TQG.B","ns":""},"count":1},"inc_marker":{"position":"00000000011.53.5","timestamp":"2020-06-04T08:33:09.476621Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000009.29.5","timestamp":"2020-06-04T08:33:09.550993Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:35.608 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-17 2020-06-04T08:33:35.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:35.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:35.609 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:35.651 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:35.652 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:35.862 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.45.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.53.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.29.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:35.863 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-17 2020-06-04T08:33:35.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:35.864 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:35.865 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:35.909 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:35.910 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:36.154 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"Qh.6cmS.CaDDDs-.d7LiIqDavhXLq3s","ns":""},"count":1},"inc_marker":{"position":"00000000004.45.5","timestamp":"2020-06-04T08:33:09.395305Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"KFtGJlLDp2kIROKHyRpH9dymw8TQG.B","ns":""},"count":1},"inc_marker":{"position":"00000000011.53.5","timestamp":"2020-06-04T08:33:09.476621Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000009.29.5","timestamp":"2020-06-04T08:33:09.550993Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:36.155 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-17 2020-06-04T08:33:36.155 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:36.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:36.157 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:36.237 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:36.237 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:36.464 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.26.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.47.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.41.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:36.465 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-17 2020-06-04T08:33:36.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:36.466 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:36.467 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:36.550 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:36.550 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:36.784 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"Qh.6cmS.CaDDDs-.d7LiIqDavhXLq3s","ns":""},"count":1},"inc_marker":{"position":"00000000004.26.5","timestamp":"2020-06-04T08:33:09.395305Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"KFtGJlLDp2kIROKHyRpH9dymw8TQG.B","ns":""},"count":1},"inc_marker":{"position":"00000000004.47.5","timestamp":"2020-06-04T08:33:09.476621Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000015.41.5","timestamp":"2020-06-04T08:33:09.550993Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:36.784 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-17 2020-06-04T08:33:36.785 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:36.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:36.786 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:36.867 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:36.867 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:37.091 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.26.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.47.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.41.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:37.092 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-17 2020-06-04T08:33:37.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:37.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:37.093 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:37.131 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:37.131 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:37.356 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"Qh.6cmS.CaDDDs-.d7LiIqDavhXLq3s","ns":""},"count":1},"inc_marker":{"position":"00000000004.26.5","timestamp":"2020-06-04T08:33:09.395305Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"KFtGJlLDp2kIROKHyRpH9dymw8TQG.B","ns":""},"count":1},"inc_marker":{"position":"00000000004.47.5","timestamp":"2020-06-04T08:33:09.476621Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"vBgqUO0Tdj66c.uXmGDy36WIk9ftlxu","ns":""},"count":1},"inc_marker":{"position":"00000000015.41.5","timestamp":"2020-06-04T08:33:09.550993Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:37.357 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-17 2020-06-04T08:33:37.357 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-17 zones={test-zone2, test-zone1} 2020-06-04T08:33:37.374 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:33:37.391 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:37.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:37.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:37.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:37.393 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:33:37.405 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:37.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:37.406 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:37.407 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:37.424 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:37.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:37.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:37.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:37.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:37.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:37.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:37.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:38.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:38.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:38.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:38.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:38.460 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-17 zones={test-zone2, test-zone1} 2020-06-04T08:33:38.461 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-17 zones={test-zone3, test-zone1} 2020-06-04T08:33:38.469 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:33:38.475 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:38.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:38.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:38.477 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:38.478 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:33:38.485 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:38.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:38.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:38.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:38.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:38.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:38.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:38.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:38.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:39.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:39.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:39.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:39.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:39.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:39.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:39.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:39.797 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-17 zones={test-zone3, test-zone1} 2020-06-04T08:33:39.799 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-17 zones={test-zone3, test-zone2} 2020-06-04T08:33:39.817 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:33:39.828 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:39.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:39.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:39.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:39.832 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:33:39.841 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:39.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:39.842 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:39.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:39.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:39.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:39.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:40.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:40.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:40.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:40.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:40.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:40.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:40.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:40.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:40.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:40.904 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-17 zones={test-zone3, test-zone2} 2020-06-04T08:33:40.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:40.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:40.906 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:40.955 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:40.956 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:41.182 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.46.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.49.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.34.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:41.182 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-18 2020-06-04T08:33:41.183 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:41.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:41.184 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:41.226 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:41.227 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:41.458 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a3aWorHM27xLX.kZUeX8g5LZc2lJmxR","ns":""},"count":1},"inc_marker":{"position":"00000000011.46.5","timestamp":"2020-06-04T08:33:09.630001Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"n8qICeGtZKx3J-ImEl0czYpahQFAdgv","ns":""},"count":1},"inc_marker":{"position":"00000000004.49.5","timestamp":"2020-06-04T08:33:09.697326Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"LThDOxUGE3zRpi0VYfsypl.WVvrhPWL","ns":""},"count":1},"inc_marker":{"position":"00000000008.34.5","timestamp":"2020-06-04T08:33:09.761590Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:41.459 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-18 2020-06-04T08:33:41.459 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:41.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:41.461 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:41.502 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:41.502 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:41.733 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.46.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.49.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.34.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:41.734 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-18 2020-06-04T08:33:41.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:41.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:41.738 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:41.819 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:41.819 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:42.058 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a3aWorHM27xLX.kZUeX8g5LZc2lJmxR","ns":""},"count":1},"inc_marker":{"position":"00000000011.46.5","timestamp":"2020-06-04T08:33:09.630001Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"n8qICeGtZKx3J-ImEl0czYpahQFAdgv","ns":""},"count":1},"inc_marker":{"position":"00000000004.49.5","timestamp":"2020-06-04T08:33:09.697326Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"LThDOxUGE3zRpi0VYfsypl.WVvrhPWL","ns":""},"count":1},"inc_marker":{"position":"00000000008.34.5","timestamp":"2020-06-04T08:33:09.761590Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:42.059 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-18 2020-06-04T08:33:42.060 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:42.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:42.062 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:42.139 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:42.155 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:42.366 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.46.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.49.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.34.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:42.367 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-18 2020-06-04T08:33:42.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:42.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:42.369 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:42.414 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:42.414 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:42.685 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.46.5","timestamp":"2020-06-04T08:33:09.630001Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.5","timestamp":"2020-06-04T08:33:09.697326Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.34.5","timestamp":"2020-06-04T08:33:09.761590Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:42.685 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-18 2020-06-04T08:33:42.686 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:42.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:42.687 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:42.767 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:42.768 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:42.979 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.26.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.47.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.33.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:42.980 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-18 2020-06-04T08:33:42.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:42.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:42.982 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:43.029 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:43.029 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:43.271 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a3aWorHM27xLX.kZUeX8g5LZc2lJmxR","ns":""},"count":1},"inc_marker":{"position":"00000000004.26.5","timestamp":"2020-06-04T08:33:09.630001Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"n8qICeGtZKx3J-ImEl0czYpahQFAdgv","ns":""},"count":2},"inc_marker":{"position":"00000000011.47.5","timestamp":"2020-06-04T08:33:09.697326Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"LThDOxUGE3zRpi0VYfsypl.WVvrhPWL","ns":""},"count":1},"inc_marker":{"position":"00000000008.33.5","timestamp":"2020-06-04T08:33:09.761590Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:43.272 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-18 2020-06-04T08:33:43.273 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:43.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:43.276 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:43.318 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:43.319 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:43.531 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.26.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.47.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.33.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:43.532 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-18 2020-06-04T08:33:43.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:43.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:43.534 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:43.580 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:43.580 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:43.800 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a3aWorHM27xLX.kZUeX8g5LZc2lJmxR","ns":""},"count":1},"inc_marker":{"position":"00000000004.26.5","timestamp":"2020-06-04T08:33:09.630001Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"n8qICeGtZKx3J-ImEl0czYpahQFAdgv","ns":""},"count":2},"inc_marker":{"position":"00000000011.47.5","timestamp":"2020-06-04T08:33:09.697326Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"LThDOxUGE3zRpi0VYfsypl.WVvrhPWL","ns":""},"count":1},"inc_marker":{"position":"00000000008.33.5","timestamp":"2020-06-04T08:33:09.761590Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:43.801 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-18 2020-06-04T08:33:43.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:43.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:43.803 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:43.846 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:43.846 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:44.079 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.25.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.28.5"},{"key":4,"val":""},{"key":5,"val":"00000000017.54.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:44.080 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-18 2020-06-04T08:33:44.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:44.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:44.082 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:44.170 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:44.171 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:44.407 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a3aWorHM27xLX.kZUeX8g5LZc2lJmxR","ns":""},"count":1},"inc_marker":{"position":"00000000004.25.5","timestamp":"2020-06-04T08:33:09.630001Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"n8qICeGtZKx3J-ImEl0czYpahQFAdgv","ns":""},"count":1},"inc_marker":{"position":"00000000004.28.5","timestamp":"2020-06-04T08:33:09.697326Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"LThDOxUGE3zRpi0VYfsypl.WVvrhPWL","ns":""},"count":1},"inc_marker":{"position":"00000000017.54.5","timestamp":"2020-06-04T08:33:09.761590Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:44.407 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-18 2020-06-04T08:33:44.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:44.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:44.409 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:44.453 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:44.454 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:44.677 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.25.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.28.5"},{"key":4,"val":""},{"key":5,"val":"00000000017.54.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:44.678 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-18 2020-06-04T08:33:44.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:44.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:44.682 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:44.726 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:44.727 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:44.957 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"a3aWorHM27xLX.kZUeX8g5LZc2lJmxR","ns":""},"count":1},"inc_marker":{"position":"00000000004.25.5","timestamp":"2020-06-04T08:33:09.630001Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"n8qICeGtZKx3J-ImEl0czYpahQFAdgv","ns":""},"count":1},"inc_marker":{"position":"00000000004.28.5","timestamp":"2020-06-04T08:33:09.697326Z"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"LThDOxUGE3zRpi0VYfsypl.WVvrhPWL","ns":""},"count":1},"inc_marker":{"position":"00000000017.54.5","timestamp":"2020-06-04T08:33:09.761590Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:44.958 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-18 2020-06-04T08:33:44.958 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-18 zones={test-zone2, test-zone1} 2020-06-04T08:33:44.972 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:33:44.980 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:44.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:44.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:44.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:44.982 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:33:44.997 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:44.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:44.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:44.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:45.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:45.024 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:45.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:45.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:45.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:45.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:45.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:45.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:45.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:45.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:45.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:45.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:46.099 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-18 zones={test-zone2, test-zone1} 2020-06-04T08:33:46.100 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-18 zones={test-zone3, test-zone1} 2020-06-04T08:33:46.109 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:33:46.117 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:46.117 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:46.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:46.118 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:46.119 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:33:46.126 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:46.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:46.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:46.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:46.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:46.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:46.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:46.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:46.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:46.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:46.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:46.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:46.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:46.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:46.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:47.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:47.183 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-18 zones={test-zone3, test-zone1} 2020-06-04T08:33:47.184 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-18 zones={test-zone3, test-zone2} 2020-06-04T08:33:47.195 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:33:47.204 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:47.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:47.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:47.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:47.207 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:33:47.213 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:33:47.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2020-06-04T08:33:47.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2020-06-04T08:33:47.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2020-06-04T08:33:47.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:47.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:47.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:33:47.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:47.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:47.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2020-06-04T08:33:47.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:47.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:47.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2020-06-04T08:33:47.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:48.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:48.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2020-06-04T08:33:48.303 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-18 zones={test-zone3, test-zone2} 2020-06-04T08:33:48.304 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2020-06-04T08:33:48.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:48.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:48.364 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:48.407 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:48.408 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:48.581 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=838ed6b2-fa2f-4917-a5c1-6c82bb405cda 2020-06-04T08:33:48.617 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259628.347410_153.1","last_update":"2020-06-04T08:33:48.347410Z"},{"marker":"1_1591259350.985454_111.1","last_update":"2020-06-04T08:29:10.985454Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:33:48.618 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:33:48.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:48.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:48.624 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:48.667 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:48.667 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:48.888 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259628.347410_153.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:33:48.347410Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:33:48.889 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:33:48.890 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:33:48.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:48.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:48.892 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:48.934 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:48.935 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:49.157 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259628.347410_153.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:33:48.347410Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:33:49.158 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:33:49.159 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:33:49.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:49.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:49.161 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:49.203 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:49.203 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:49.470 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259628.347410_153.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:33:48.347410Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:33:49.471 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:33:49.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:49.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:49.528 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:49.573 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:49.574 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:49.778 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.15.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:49.779 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-19 2020-06-04T08:33:49.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:49.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:49.781 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:49.833 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:49.834 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:50.052 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000002.15.10","timestamp":"2020-06-04T08:33:49.679483Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:50.053 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-19 2020-06-04T08:33:50.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:50.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:50.056 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:50.113 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:50.115 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:50.318 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.21.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:50.319 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-19 2020-06-04T08:33:50.320 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:50.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:50.321 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:50.369 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:50.370 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:50.616 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000002.15.10","timestamp":"2020-06-04T08:33:49.669451Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:50.616 WARNING:rgw_multi.tests:bucket gpxthc-19 zone test-zone3 behind zone test-zone1: shard=5 master=00000000005.21.9 target=00000000002.15.10 2020-06-04T08:33:55.618 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:55.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:55.620 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:55.668 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:55.668 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:55.914 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000005.21.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:55.915 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-19 2020-06-04T08:33:55.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:55.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:55.917 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:55.959 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:55.959 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:56.188 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000006.23.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:56.189 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-19 2020-06-04T08:33:56.190 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:56.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:56.192 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:56.237 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:56.237 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:56.607 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.23.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:56.608 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-19 2020-06-04T08:33:56.609 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:56.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:56.610 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:56.691 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:56.692 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:56.910 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.21.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:56.911 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-19 2020-06-04T08:33:56.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:56.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:56.913 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:56.958 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:56.958 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:57.203 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000004.21.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:57.204 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-19 2020-06-04T08:33:57.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:57.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:57.205 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:57.252 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:57.252 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:57.465 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.21.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:57.466 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-19 2020-06-04T08:33:57.467 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:57.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:57.469 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:57.514 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:57.569 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:57.762 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000004.21.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:57.763 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-19 2020-06-04T08:33:57.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:57.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:57.765 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:57.847 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:57.848 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:58.076 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:33:58.076 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-19 2020-06-04T08:33:58.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:58.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:33:58.078 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:33:58.162 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:33:58.163 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:33:58.407 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000003.20.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:33:58.408 WARNING:rgw_multi.tests:bucket gpxthc-19 zone test-zone1 behind zone test-zone3: shard=5 master=00000000004.22.9 target=00000000003.20.9 2020-06-04T08:34:03.409 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:03.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:03.412 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:34:03.467 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:34:03.467 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:34:03.715 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000003.20.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:34:03.715 WARNING:rgw_multi.tests:bucket gpxthc-19 zone test-zone1 behind zone test-zone3: shard=5 master=00000000004.22.9 target=00000000003.20.9 2020-06-04T08:34:08.717 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:08.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:08.719 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:34:08.769 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:34:08.770 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:34:09.005 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000003.20.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:34:09.006 WARNING:rgw_multi.tests:bucket gpxthc-19 zone test-zone1 behind zone test-zone3: shard=5 master=00000000004.22.9 target=00000000003.20.9 2020-06-04T08:34:14.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:14.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:14.009 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:34:14.059 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:34:14.060 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:34:14.302 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:34:14.303 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-19 2020-06-04T08:34:14.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:14.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:14.306 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:34:14.384 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:34:14.384 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:34:14.612 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:34:14.613 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-19 2020-06-04T08:34:14.614 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:14.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:14.616 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:34:14.663 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:34:14.664 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:34:14.903 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:34:14.903 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-19 2020-06-04T08:34:14.904 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-19 zones={test-zone2, test-zone1} 2020-06-04T08:34:14.916 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:34:14.967 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:34:14.968 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:34:14.986 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:34:15.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:34:15.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:34:15.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:34:15.202 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-19 zones={test-zone2, test-zone1} 2020-06-04T08:34:15.203 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-19 zones={test-zone3, test-zone1} 2020-06-04T08:34:15.213 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:34:15.268 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:34:15.269 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:34:15.280 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:34:15.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:34:15.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:34:15.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:34:15.493 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-19 zones={test-zone3, test-zone1} 2020-06-04T08:34:15.493 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-19 zones={test-zone3, test-zone2} 2020-06-04T08:34:15.512 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:34:15.562 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:34:15.563 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:34:15.572 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:34:15.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:34:15.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:34:15.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:34:15.790 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-19 zones={test-zone3, test-zone2} 2020-06-04T08:34:17.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:17.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:17.361 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:34:17.403 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:34:17.404 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:34:17.624 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000006.23.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000027.101.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:34:17.625 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-19 2020-06-04T08:34:17.625 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:17.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:17.627 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:34:17.672 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:34:17.672 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:34:17.911 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000006.23.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"WHWwhBgObP7aCYZlfBx11EuVUSaYmmJ","ns":""},"count":1},"inc_marker":{"position":"00000000004.48.11","timestamp":"2020-06-04T08:34:15.953149Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:34:17.911 WARNING:rgw_multi.tests:bucket gpxthc-19 zone test-zone2 behind zone test-zone1: shard=8 master=00000000027.101.11 target=00000000004.48.11 2020-06-04T08:34:22.918 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:22.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:22.920 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:34:22.963 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:34:22.963 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:34:23.177 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000006.23.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"WHWwhBgObP7aCYZlfBx11EuVUSaYmmJ","ns":""},"count":1},"inc_marker":{"position":"00000000004.48.11","timestamp":"2020-06-04T08:34:15.953149Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:34:23.178 WARNING:rgw_multi.tests:bucket gpxthc-19 zone test-zone2 behind zone test-zone1: shard=8 master=00000000027.101.11 target=00000000004.48.11 2020-06-04T08:34:28.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:28.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:28.181 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:34:28.230 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:34:28.230 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:34:28.463 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000006.23.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"WHWwhBgObP7aCYZlfBx11EuVUSaYmmJ","ns":""},"count":1},"inc_marker":{"position":"00000000004.48.11","timestamp":"2020-06-04T08:34:15.953149Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:34:28.464 WARNING:rgw_multi.tests:bucket gpxthc-19 zone test-zone2 behind zone test-zone1: shard=8 master=00000000027.101.11 target=00000000004.48.11 2020-06-04T08:34:33.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:33.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:33.468 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:34:33.514 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:34:33.515 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:34:33.749 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000006.23.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"WHWwhBgObP7aCYZlfBx11EuVUSaYmmJ","ns":""},"count":1},"inc_marker":{"position":"00000000004.48.11","timestamp":"2020-06-04T08:34:15.953149Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:34:33.749 WARNING:rgw_multi.tests:bucket gpxthc-19 zone test-zone2 behind zone test-zone1: shard=8 master=00000000027.101.11 target=00000000004.48.11 2020-06-04T08:34:38.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:38.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:38.754 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:34:38.800 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:34:38.800 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:34:39.035 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000006.23.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"WHWwhBgObP7aCYZlfBx11EuVUSaYmmJ","ns":""},"count":1},"inc_marker":{"position":"00000000030.112.5","timestamp":"2020-06-04T08:34:16.471008Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:34:39.036 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-19 2020-06-04T08:34:39.037 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:39.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:39.039 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:34:39.084 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:34:39.085 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:34:39.303 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000006.23.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000048.159.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:34:39.304 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-19 2020-06-04T08:34:39.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:39.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:39.306 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:34:39.392 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:34:39.392 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:34:39.633 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000006.23.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"WHWwhBgObP7aCYZlfBx11EuVUSaYmmJ","ns":""},"count":1},"inc_marker":{"position":"00000000045.153.5","timestamp":"2020-06-04T08:34:17.160138Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:34:39.634 WARNING:rgw_multi.tests:bucket gpxthc-19 zone test-zone3 behind zone test-zone1: shard=8 master=00000000048.159.9 target=00000000045.153.5 2020-06-04T08:34:44.636 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:44.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:44.640 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:34:44.689 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:34:44.689 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:34:44.926 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000006.23.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"WHWwhBgObP7aCYZlfBx11EuVUSaYmmJ","ns":""},"count":1},"inc_marker":{"position":"00000000045.153.5","timestamp":"2020-06-04T08:34:17.160138Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:34:44.926 WARNING:rgw_multi.tests:bucket gpxthc-19 zone test-zone3 behind zone test-zone1: shard=8 master=00000000048.159.9 target=00000000045.153.5 2020-06-04T08:34:49.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:49.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:49.930 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:34:49.974 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:34:49.974 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:34:50.190 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000006.23.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"WHWwhBgObP7aCYZlfBx11EuVUSaYmmJ","ns":""},"count":1},"inc_marker":{"position":"00000000045.153.5","timestamp":"2020-06-04T08:34:17.160138Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:34:50.191 WARNING:rgw_multi.tests:bucket gpxthc-19 zone test-zone3 behind zone test-zone1: shard=8 master=00000000048.159.9 target=00000000045.153.5 2020-06-04T08:34:55.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:55.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:34:55.194 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:34:55.238 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:34:55.239 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:34:55.492 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000006.23.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"WHWwhBgObP7aCYZlfBx11EuVUSaYmmJ","ns":""},"count":1},"inc_marker":{"position":"00000000045.153.5","timestamp":"2020-06-04T08:34:17.160138Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:34:55.492 WARNING:rgw_multi.tests:bucket gpxthc-19 zone test-zone3 behind zone test-zone1: shard=8 master=00000000048.159.9 target=00000000045.153.5 2020-06-04T08:35:00.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:00.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:00.496 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:35:00.542 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:35:00.543 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:35:00.773 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000006.23.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"WHWwhBgObP7aCYZlfBx11EuVUSaYmmJ","ns":""},"count":1},"inc_marker":{"position":"00000000048.159.9","timestamp":"2020-06-04T08:34:17.261683Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:35:00.773 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-19 2020-06-04T08:35:00.774 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:00.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:00.775 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:35:00.852 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:35:00.853 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:35:01.072 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000006.23.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000049.161.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:35:01.074 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-19 2020-06-04T08:35:01.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:01.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:01.075 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:35:01.124 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:35:01.125 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:35:01.391 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.23.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000048.159.9","timestamp":"2020-06-04T08:34:17.261683Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:35:01.391 WARNING:rgw_multi.tests:bucket gpxthc-19 zone test-zone4 behind zone test-zone1: shard=8 master=00000000049.161.9 target=00000000048.159.9 2020-06-04T08:35:06.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:06.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:06.395 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:35:06.442 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:35:06.442 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:35:06.710 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.23.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000048.159.9","timestamp":"2020-06-04T08:34:17.261683Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:35:06.710 WARNING:rgw_multi.tests:bucket gpxthc-19 zone test-zone4 behind zone test-zone1: shard=8 master=00000000049.161.9 target=00000000048.159.9 2020-06-04T08:35:11.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:11.713 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:11.714 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:35:11.764 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:35:11.764 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:35:12.027 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.23.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000048.159.9","timestamp":"2020-06-04T08:34:17.261683Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:35:12.028 WARNING:rgw_multi.tests:bucket gpxthc-19 zone test-zone4 behind zone test-zone1: shard=8 master=00000000049.161.9 target=00000000048.159.9 2020-06-04T08:35:17.029 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:17.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:17.031 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:35:17.075 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:35:17.087 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:35:17.336 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.23.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000048.159.9","timestamp":"2020-06-04T08:34:17.261683Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:35:17.337 WARNING:rgw_multi.tests:bucket gpxthc-19 zone test-zone4 behind zone test-zone1: shard=8 master=00000000049.161.9 target=00000000048.159.9 2020-06-04T08:35:22.339 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:22.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:22.341 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:35:22.386 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:35:22.386 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:35:22.656 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.23.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000048.159.9","timestamp":"2020-06-04T08:34:17.261683Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:35:22.657 WARNING:rgw_multi.tests:bucket gpxthc-19 zone test-zone4 behind zone test-zone1: shard=8 master=00000000049.161.9 target=00000000048.159.9 2020-06-04T08:35:27.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:27.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:27.663 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:35:27.707 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:35:27.707 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:35:27.978 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.23.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000048.159.9","timestamp":"2020-06-04T08:34:17.261683Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:35:27.978 WARNING:rgw_multi.tests:bucket gpxthc-19 zone test-zone4 behind zone test-zone1: shard=8 master=00000000049.161.9 target=00000000048.159.9 2020-06-04T08:35:32.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:32.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:32.982 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:35:33.028 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:35:33.029 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:35:33.299 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.23.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000049.161.9","timestamp":"2020-06-04T08:34:17.313762Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:35:33.300 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-19 2020-06-04T08:35:33.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:33.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:33.302 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:35:33.388 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:35:33.389 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:35:33.597 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.21.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000044.165.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:35:33.598 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-19 2020-06-04T08:35:33.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:33.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:33.600 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:35:33.644 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:35:33.645 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:35:33.874 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000004.21.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"yJRTYwU2.jL6tYD3QFyYFBLkFnMj0wo","ns":""},"count":5},"inc_marker":{"position":"00000000044.165.9","timestamp":"2020-06-04T08:34:17.313762Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:35:33.874 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-19 2020-06-04T08:35:33.875 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:33.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:33.876 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:35:33.927 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:35:33.927 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:35:34.137 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.21.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000044.165.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:35:34.138 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-19 2020-06-04T08:35:34.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:34.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:34.139 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:35:34.186 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:35:34.187 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:35:34.435 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000004.21.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"yJRTYwU2.jL6tYD3QFyYFBLkFnMj0wo","ns":""},"count":5},"inc_marker":{"position":"00000000044.165.9","timestamp":"2020-06-04T08:34:17.313762Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:35:34.436 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-19 2020-06-04T08:35:34.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:34.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:34.437 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:35:34.477 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:35:34.478 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:35:34.699 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000047.166.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:35:34.700 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-19 2020-06-04T08:35:34.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:34.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:34.704 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:35:34.786 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:35:34.786 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:35:35.027 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"AxtKBHPnFnOUXospjapnz3pVjEHtdvg","ns":""},"count":1},"inc_marker":{"position":"00000000047.166.9","timestamp":"2020-06-04T08:34:17.261683Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:35:35.027 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-19 2020-06-04T08:35:35.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:35.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:35.029 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:35:35.070 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:35:35.070 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:35:35.302 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000047.166.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:35:35.303 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-19 2020-06-04T08:35:35.304 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:35.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:35:35.305 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:35:35.350 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:35:35.350 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:35:35.592 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"4tFQFJLuWEyrbnqCX6nnpqwsN0DRaY5","ns":""},"count":1},"inc_marker":{"position":"00000000004.22.9","timestamp":"2020-06-04T08:33:49.480622Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"AxtKBHPnFnOUXospjapnz3pVjEHtdvg","ns":""},"count":1},"inc_marker":{"position":"00000000047.166.9","timestamp":"2020-06-04T08:34:17.261683Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:35:35.592 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-19 2020-06-04T08:35:35.593 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-19 zones={test-zone2, test-zone1} 2020-06-04T08:35:35.623 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:35:35.638 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:35:35.639 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.640 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.640 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.642 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.646 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.648 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.648 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.649 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.649 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.653 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.655 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.656 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.658 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:35:35.672 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:35:35.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.676 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.680 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.680 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.680 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.685 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.688 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.689 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.689 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:35.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:35:35.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:35:35.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:35:35.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:35.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:36.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:36.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:36.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:36.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:36.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:36.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:36.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:36.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:36.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:37.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:37.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:37.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:37.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:37.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:37.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:37.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:37.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:37.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:37.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:38.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:38.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:38.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:38.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:38.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:38.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:38.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:38.688 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:38.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:38.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:38.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:39.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:39.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:39.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:39.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:39.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:39.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:39.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:39.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:39.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:40.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:40.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:40.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:40.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:40.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:40.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:40.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:40.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:40.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:40.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:41.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:41.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:41.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:41.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:41.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:41.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:41.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:41.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:41.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:41.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:42.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:42.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:42.261 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:42.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:42.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:42.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:42.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:42.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:42.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:42.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:43.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:43.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:43.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:43.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:43.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:43.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:43.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:43.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:43.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:43.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:44.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:44.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:44.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:44.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:44.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:44.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:44.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:44.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:44.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:44.969 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-19 zones={test-zone2, test-zone1} 2020-06-04T08:35:44.970 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-19 zones={test-zone3, test-zone1} 2020-06-04T08:35:44.980 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:35:44.990 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:35:44.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:44.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:44.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:44.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:44.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:44.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:44.993 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:44.993 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:44.994 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:44.994 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:44.995 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:44.995 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:44.995 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:44.996 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:44.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:44.997 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:44.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:44.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:44.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:44.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.001 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.002 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.004 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:35:45.015 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:35:45.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.016 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.017 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.018 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.018 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.019 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.020 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.020 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.020 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.021 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.022 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.022 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.022 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.027 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:45.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:35:45.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:35:45.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:35:45.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:45.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:45.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:45.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:45.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:45.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:45.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:45.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:45.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:46.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:46.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:46.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:46.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:46.460 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:46.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:46.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:46.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:46.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:47.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:47.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:47.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:47.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:47.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:47.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:47.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:47.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:47.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:47.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:48.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:48.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:48.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:48.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:48.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:48.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:48.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:48.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:48.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:48.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:49.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:49.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:49.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:49.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:49.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:49.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:49.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:49.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:49.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:49.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:50.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:50.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:50.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:50.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:50.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:50.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:50.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:50.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:50.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:50.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:50.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:51.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:51.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:51.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:51.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:51.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:51.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:51.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:51.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:51.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:51.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:52.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:52.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:52.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:52.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:52.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:52.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:52.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:52.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:52.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:53.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:53.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:53.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:53.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:53.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:53.484 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:53.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:53.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:53.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:53.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:53.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:54.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:54.201 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-19 zones={test-zone3, test-zone1} 2020-06-04T08:35:54.202 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-19 zones={test-zone3, test-zone2} 2020-06-04T08:35:54.213 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:35:54.226 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:35:54.226 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.227 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.228 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.232 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.233 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.236 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.239 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.240 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.241 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.241 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.242 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.243 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.244 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.245 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.246 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:35:54.257 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2020-06-04T08:35:54.257 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.263 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.266 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.269 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.270 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.277 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.278 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:35:54.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:35:54.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:35:54.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2020-06-04T08:35:54.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:54.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:54.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:54.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:54.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:54.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:55.079 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:55.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:55.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:55.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:55.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:55.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:55.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:55.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:55.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:56.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:56.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:56.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:56.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:56.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:56.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:56.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:56.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:56.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:56.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:57.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:57.178 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:57.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:57.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:57.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:57.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:57.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:57.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:57.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:57.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:58.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:58.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:58.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:58.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:58.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:58.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:58.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:58.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:58.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:59.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:59.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:59.216 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:59.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:59.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:59.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:59.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:59.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:59.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:35:59.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:00.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:00.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:00.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:00.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:00.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:00.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:00.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:00.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:00.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:00.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:01.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:01.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:01.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:01.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:01.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:01.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:01.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:01.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:01.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:01.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:02.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:02.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:02.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:02.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:02.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:02.593 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:02.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:02.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:02.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:03.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:03.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:03.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:03.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:03.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:03.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:03.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:03.763 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-19 zones={test-zone3, test-zone2} 2020-06-04T08:36:03.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:03.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:03.771 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:03.814 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:03.815 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:04.052 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259652.395954_88.1","last_update":"2020-06-04T08:34:12.395954Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1591259589.882369_86.1","last_update":"2020-06-04T08:33:09.882369Z"},{"marker":"1_1591259718.410430_174.1","last_update":"2020-06-04T08:35:18.410430Z"}] 2020-06-04T08:36:04.053 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-06-04T08:36:04.054 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:04.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:04.056 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:04.097 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:04.097 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:04.304 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11207024744551397021},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591259652.395954_88.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:34:12.395954Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591259718.410430_174.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:35:18.410430Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:36:04.304 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-06-04T08:36:04.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:04.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:04.307 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:04.356 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:04.357 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:04.589 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259652.395954_88.1","last_update":"2020-06-04T08:34:12.395954Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1591259589.882369_86.1","last_update":"2020-06-04T08:33:09.882369Z"},{"marker":"1_1591259718.410430_174.1","last_update":"2020-06-04T08:35:18.410430Z"}] 2020-06-04T08:36:04.590 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-06-04T08:36:04.590 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:04.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:04.592 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:04.678 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:04.679 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:04.895 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2627143167498148538},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591259652.395954_88.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:34:12.395954Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591259718.410430_174.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:35:18.410430Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:36:04.896 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-06-04T08:36:04.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:04.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:04.898 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:04.936 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:04.936 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:05.169 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259652.395954_88.1","last_update":"2020-06-04T08:34:12.395954Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1591259589.882369_86.1","last_update":"2020-06-04T08:33:09.882369Z"},{"marker":"1_1591259718.410430_174.1","last_update":"2020-06-04T08:35:18.410430Z"}] 2020-06-04T08:36:05.170 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-06-04T08:36:05.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:05.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:05.172 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:05.211 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:05.211 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:05.479 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7644369918941386260},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591259652.395954_88.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:34:12.395954Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591259718.410430_174.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:35:18.410430Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:36:05.480 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-06-04T08:36:05.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:05.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:05.482 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:05.559 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:05.560 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:05.781 INFO:teuthology.orchestra.run.smithi196.stdout:[{"marker":"1_1591259652.539635_76.1","last_update":"2020-06-04T08:34:12.539635Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1591259589.701718_107.1","last_update":"2020-06-04T08:33:09.701718Z"},{"marker":"1_1591259696.548298_186.1","last_update":"2020-06-04T08:34:56.548298Z"}] 2020-06-04T08:36:05.781 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-06-04T08:36:05.782 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:05.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:05.783 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:05.828 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:05.828 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:06.062 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1308327179334993417},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591259652.539635_76.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:34:12.539635Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.701718_107.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.701718Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591259696.548298_186.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:34:56.548298Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:36:06.093 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-06-04T08:36:06.093 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:06.094 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:06.095 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:06.135 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:06.135 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:06.352 INFO:teuthology.orchestra.run.smithi196.stdout:[{"marker":"1_1591259652.539635_76.1","last_update":"2020-06-04T08:34:12.539635Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1591259589.701718_107.1","last_update":"2020-06-04T08:33:09.701718Z"},{"marker":"1_1591259696.548298_186.1","last_update":"2020-06-04T08:34:56.548298Z"}] 2020-06-04T08:36:06.352 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-06-04T08:36:06.353 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:06.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:06.354 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:06.401 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:06.402 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:06.626 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":16507416453976119466},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591259652.539635_76.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:34:12.539635Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.701718_107.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.701718Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591259696.548298_186.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:34:56.548298Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:36:06.627 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-06-04T08:36:06.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:06.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:06.629 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:06.707 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:06.708 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:06.920 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259652.444513_63.1","last_update":"2020-06-04T08:34:12.444513Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1591259589.886327_75.1","last_update":"2020-06-04T08:33:09.886327Z"},{"marker":"1_1591259696.454795_89.1","last_update":"2020-06-04T08:34:56.454795Z"}] 2020-06-04T08:36:06.920 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-06-04T08:36:06.921 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:06.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:06.923 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:07.005 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:07.005 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:07.230 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3830535810264008500},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591259652.444513_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:34:12.444513Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.886327_75.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.886327Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591259696.454795_89.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:34:56.454795Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:36:07.231 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-06-04T08:36:07.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:07.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:07.232 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:07.315 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:07.316 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:07.548 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259652.444513_63.1","last_update":"2020-06-04T08:34:12.444513Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1591259589.886327_75.1","last_update":"2020-06-04T08:33:09.886327Z"},{"marker":"1_1591259696.454795_89.1","last_update":"2020-06-04T08:34:56.454795Z"}] 2020-06-04T08:36:07.548 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-06-04T08:36:07.549 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:07.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:07.552 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:07.595 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:07.596 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:07.822 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9078143951431167771},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591259652.444513_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:34:12.444513Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.886327_75.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.886327Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591259696.454795_89.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:34:56.454795Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:36:07.823 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-06-04T08:36:07.824 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2020-06-04T08:36:07.860 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:07.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:07.862 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:07.899 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:07.899 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:08.070 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=838ed6b2-fa2f-4917-a5c1-6c82bb405cda 2020-06-04T08:36:08.106 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259767.852808_747.1","last_update":"2020-06-04T08:36:07.852808Z"},{"marker":"1_1591259350.985454_111.1","last_update":"2020-06-04T08:29:10.985454Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:36:08.107 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:36:08.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:08.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:08.108 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:08.149 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:08.150 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:08.370 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259767.852808_747.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:36:07.852808Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:36:08.371 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:36:08.371 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:36:08.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:08.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:08.373 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:08.422 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:08.422 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:08.648 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259767.852808_747.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:36:07.852808Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:36:08.649 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:36:08.650 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:36:08.650 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:08.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:08.651 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:08.696 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:08.697 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:08.973 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259767.852808_747.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:36:07.852808Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:36:08.974 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:36:09.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:09.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:09.032 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:09.072 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:09.072 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:09.283 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.26.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:36:09.284 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-20 2020-06-04T08:36:09.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:09.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:09.287 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:09.327 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:09.328 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:09.545 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.26.5","timestamp":"2020-06-04T08:36:09.163520Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:36:09.546 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-20 2020-06-04T08:36:09.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:09.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:09.548 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:09.587 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:09.588 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:09.808 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.26.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:36:09.808 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-20 2020-06-04T08:36:09.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:09.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:09.810 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:09.892 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:09.892 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:10.140 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.26.5","timestamp":"2020-06-04T08:36:09.161423Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:36:10.140 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-20 2020-06-04T08:36:10.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:10.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:10.147 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:10.227 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:10.228 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:10.452 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.26.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:36:10.453 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-20 2020-06-04T08:36:10.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:10.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:10.455 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:10.501 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:10.502 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:10.776 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.26.5","timestamp":"2020-06-04T08:36:08.981194Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:36:10.777 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-20 2020-06-04T08:36:10.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:10.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:10.778 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:10.857 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:10.857 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:11.075 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.46.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:36:11.076 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-20 2020-06-04T08:36:11.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:11.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:11.078 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:11.123 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:11.123 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:11.382 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5","timestamp":"2020-06-04T08:36:09.357493Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:36:11.383 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-20 2020-06-04T08:36:11.383 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:11.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:11.385 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:11.426 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:11.427 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:11.660 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.46.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:36:11.661 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-20 2020-06-04T08:36:11.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:11.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:11.662 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:11.709 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:11.710 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:11.932 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5","timestamp":"2020-06-04T08:36:09.361509Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:36:11.933 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-20 2020-06-04T08:36:11.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:11.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:11.936 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:11.989 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:11.990 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:12.204 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.26.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:36:12.205 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-20 2020-06-04T08:36:12.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:12.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:12.207 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:12.293 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:12.293 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:12.913 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.26.5","timestamp":"2020-06-04T08:36:09.441497Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:36:12.914 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-20 2020-06-04T08:36:12.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:12.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:12.917 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:12.961 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:12.961 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:13.187 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.26.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:36:13.187 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-20 2020-06-04T08:36:13.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:13.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:13.189 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:13.233 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:13.234 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:13.477 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.26.5","timestamp":"2020-06-04T08:36:09.447446Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:36:13.478 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-20 2020-06-04T08:36:13.478 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-20 zones={test-zone2, test-zone1} 2020-06-04T08:36:13.504 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:36:13.554 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:36:13.555 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:36:13.564 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:36:13.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:13.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:13.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:13.791 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-20 zones={test-zone2, test-zone1} 2020-06-04T08:36:13.791 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-20 zones={test-zone3, test-zone1} 2020-06-04T08:36:13.801 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:36:13.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:36:13.862 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:36:13.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:36:13.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:13.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:14.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:14.078 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-20 zones={test-zone3, test-zone1} 2020-06-04T08:36:14.079 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-20 zones={test-zone3, test-zone2} 2020-06-04T08:36:14.091 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:36:14.138 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:36:14.139 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:36:14.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:36:14.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:14.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:14.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:36:14.353 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-20 zones={test-zone3, test-zone2} 2020-06-04T08:36:14.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:14.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:14.373 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:14.411 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:14.412 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:14.590 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=838ed6b2-fa2f-4917-a5c1-6c82bb405cda 2020-06-04T08:36:14.630 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259774.363917_755.1","last_update":"2020-06-04T08:36:14.363917Z"},{"marker":"1_1591259350.985454_111.1","last_update":"2020-06-04T08:29:10.985454Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:36:14.631 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:36:14.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:14.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:14.633 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:14.676 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:14.677 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:14.906 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259774.363917_755.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:36:14.363917Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:36:14.906 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:36:14.907 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:36:14.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:14.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:14.909 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:14.950 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:14.950 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:15.172 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259774.363917_755.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:36:14.363917Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:36:15.173 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:36:15.173 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:36:15.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:15.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:15.175 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:15.221 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:15.222 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:15.490 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259774.363917_755.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:36:14.363917Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:36:15.491 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:36:15.542 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:15.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:15.545 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:15.586 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:15.586 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:15.809 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.29.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:36:15.810 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-20 2020-06-04T08:36:15.811 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:15.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:15.812 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:15.858 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:15.859 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:16.095 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.26.5","timestamp":"2020-06-04T08:36:09.163520Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:36:16.096 WARNING:rgw_multi.tests:bucket gpxthc-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000003.29.12 target=00000000001.26.5 2020-06-04T08:36:21.097 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:21.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:21.099 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:21.144 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:21.145 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:21.378 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.26.5","timestamp":"2020-06-04T08:36:09.163520Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:36:21.379 WARNING:rgw_multi.tests:bucket gpxthc-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000003.29.12 target=00000000001.26.5 2020-06-04T08:36:26.381 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:26.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:26.384 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:26.429 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:26.430 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:26.668 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.26.5","timestamp":"2020-06-04T08:36:09.163520Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:36:26.669 WARNING:rgw_multi.tests:bucket gpxthc-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000003.29.12 target=00000000001.26.5 2020-06-04T08:36:31.671 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:31.672 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:31.673 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:31.728 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:31.729 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:31.964 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.29.12","timestamp":"2020-06-04T08:36:15.498741Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:36:31.964 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-20 2020-06-04T08:36:31.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:31.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:31.966 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:32.010 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:32.010 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:32.235 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.29.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:36:32.236 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-20 2020-06-04T08:36:32.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:32.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:32.238 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:32.322 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:32.323 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:32.576 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.29.12","timestamp":"2020-06-04T08:36:15.498741Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:36:32.576 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-20 2020-06-04T08:36:32.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:32.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:32.578 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:32.662 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:32.662 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:32.886 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.29.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:36:32.887 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-20 2020-06-04T08:36:32.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:32.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:32.889 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:32.934 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:32.934 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:33.211 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.29.12","timestamp":"2020-06-04T08:36:15.498741Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:36:33.211 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-20 2020-06-04T08:36:33.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:33.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:33.213 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:33.292 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:33.293 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:33.516 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.50.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:36:33.517 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-20 2020-06-04T08:36:33.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:33.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:33.519 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:33.565 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:33.565 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:33.798 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5","timestamp":"2020-06-04T08:36:09.357493Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:36:33.799 WARNING:rgw_multi.tests:bucket gpxthc-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000003.50.12 target=00000000001.46.5 2020-06-04T08:36:38.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:38.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:38.812 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:38.857 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:38.857 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:39.112 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5","timestamp":"2020-06-04T08:36:09.357493Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:36:39.112 WARNING:rgw_multi.tests:bucket gpxthc-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000003.50.12 target=00000000001.46.5 2020-06-04T08:36:44.115 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:44.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:44.117 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:44.163 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:44.163 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:44.416 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5","timestamp":"2020-06-04T08:36:09.357493Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:36:44.417 WARNING:rgw_multi.tests:bucket gpxthc-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000003.50.12 target=00000000001.46.5 2020-06-04T08:36:49.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:49.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:49.420 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:49.460 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:49.461 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:49.685 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.50.12","timestamp":"2020-06-04T08:36:15.498741Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:36:49.686 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-20 2020-06-04T08:36:49.687 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:49.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:49.688 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:49.730 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:49.730 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:49.948 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.52.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:36:49.948 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-20 2020-06-04T08:36:49.949 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:49.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:49.950 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:49.996 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:49.997 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:50.210 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.50.12","timestamp":"2020-06-04T08:36:15.498741Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:36:50.211 WARNING:rgw_multi.tests:bucket gpxthc-20 zone test-zone3 behind zone test-zone2: shard=8 master=00000000004.52.9 target=00000000003.50.12 2020-06-04T08:36:55.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:55.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:36:55.214 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:36:55.262 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:36:55.263 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:36:55.517 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.50.12","timestamp":"2020-06-04T08:36:15.498741Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:36:55.518 WARNING:rgw_multi.tests:bucket gpxthc-20 zone test-zone3 behind zone test-zone2: shard=8 master=00000000004.52.9 target=00000000003.50.12 2020-06-04T08:37:00.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:00.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:00.523 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:00.569 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:00.570 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:00.806 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.50.12","timestamp":"2020-06-04T08:36:15.498741Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:37:00.807 WARNING:rgw_multi.tests:bucket gpxthc-20 zone test-zone3 behind zone test-zone2: shard=8 master=00000000004.52.9 target=00000000003.50.12 2020-06-04T08:37:05.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:05.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:05.814 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:05.871 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:05.871 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:06.133 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.50.12","timestamp":"2020-06-04T08:36:15.498741Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:37:06.134 WARNING:rgw_multi.tests:bucket gpxthc-20 zone test-zone3 behind zone test-zone2: shard=8 master=00000000004.52.9 target=00000000003.50.12 2020-06-04T08:37:11.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:11.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:11.139 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:11.188 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:11.188 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:11.437 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.52.9","timestamp":"2020-06-04T08:36:15.498741Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:37:11.438 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-20 2020-06-04T08:37:11.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:11.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:11.441 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:11.521 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:11.521 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:11.738 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.32.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:37:11.738 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-20 2020-06-04T08:37:11.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:11.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:11.740 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:11.824 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:11.825 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:12.061 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.32.9","timestamp":"2020-06-04T08:36:15.498741Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:37:12.062 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-20 2020-06-04T08:37:12.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:12.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:12.064 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:12.144 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:12.145 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:12.382 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.32.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:37:12.383 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-20 2020-06-04T08:37:12.384 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:12.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:12.385 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:12.438 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:12.438 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:12.676 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.32.9","timestamp":"2020-06-04T08:36:15.498741Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:37:12.676 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-20 2020-06-04T08:37:12.677 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-20 zones={test-zone2, test-zone1} 2020-06-04T08:37:12.699 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:37:12.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:12.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:12.749 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:37:12.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:12.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:12.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:12.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:12.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:12.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:12.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:13.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:13.142 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-20 zones={test-zone2, test-zone1} 2020-06-04T08:37:13.143 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-20 zones={test-zone3, test-zone1} 2020-06-04T08:37:13.153 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:37:13.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:13.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:13.207 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:37:13.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:13.221 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:13.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:13.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:13.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:13.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:13.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:13.510 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:13.572 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-20 zones={test-zone3, test-zone1} 2020-06-04T08:37:13.572 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-20 zones={test-zone3, test-zone2} 2020-06-04T08:37:13.583 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:37:13.634 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:13.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:13.636 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:37:13.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:13.645 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:13.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:13.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:13.781 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:13.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:13.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:13.930 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:13.993 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-20 zones={test-zone3, test-zone2} 2020-06-04T08:37:14.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:14.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:14.017 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:14.078 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:14.078 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:14.252 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=838ed6b2-fa2f-4917-a5c1-6c82bb405cda 2020-06-04T08:37:14.290 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259834.004633_772.1","last_update":"2020-06-04T08:37:14.004633Z"},{"marker":"1_1591259350.985454_111.1","last_update":"2020-06-04T08:29:10.985454Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:37:14.290 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:37:14.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:14.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:14.292 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:14.334 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:14.334 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:14.562 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259834.004633_772.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:37:14.004633Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:37:14.563 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:37:14.564 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:37:14.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:14.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:14.566 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:14.611 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:14.612 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:14.843 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259834.004633_772.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:37:14.004633Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:37:14.844 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:37:14.844 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:37:14.845 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:14.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:14.846 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:14.888 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:14.888 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:15.164 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259834.004633_772.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:37:14.004633Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:37:15.164 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:37:15.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:15.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:15.224 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:15.265 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:15.265 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:15.487 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.33.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:37:15.487 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-20 2020-06-04T08:37:15.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:15.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:15.490 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:15.535 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:15.535 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:15.771 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000005.33.17","timestamp":"2020-06-04T08:37:15.173082Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:37:15.772 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-20 2020-06-04T08:37:15.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:15.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:15.775 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:15.822 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:15.822 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:16.050 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.33.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:37:16.050 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-20 2020-06-04T08:37:16.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:16.051 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:16.052 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:16.095 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:16.095 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:16.345 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000005.33.17","timestamp":"2020-06-04T08:37:15.173082Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:37:16.345 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-20 2020-06-04T08:37:16.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:16.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:16.347 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:16.432 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:16.432 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:16.663 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.33.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:37:16.664 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-20 2020-06-04T08:37:16.664 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:16.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:16.665 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:16.709 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:16.709 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:16.978 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.33.17","timestamp":"2020-06-04T08:37:15.173082Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:37:16.979 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-20 2020-06-04T08:37:16.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:16.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:16.983 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:17.060 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:17.061 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:17.283 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000006.57.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:37:17.283 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-20 2020-06-04T08:37:17.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:17.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:17.288 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:17.335 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:17.336 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:17.589 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.52.9","timestamp":"2020-06-04T08:36:15.498741Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:37:17.590 WARNING:rgw_multi.tests:bucket gpxthc-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000006.57.17 target=00000000004.52.9 2020-06-04T08:37:22.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:22.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:22.593 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:22.640 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:22.641 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:22.872 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.52.9","timestamp":"2020-06-04T08:36:15.498741Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:37:22.873 WARNING:rgw_multi.tests:bucket gpxthc-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000006.57.17 target=00000000004.52.9 2020-06-04T08:37:27.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:27.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:27.876 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:27.921 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:27.922 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:28.163 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.52.9","timestamp":"2020-06-04T08:36:15.498741Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:37:28.164 WARNING:rgw_multi.tests:bucket gpxthc-20 zone test-zone1 behind zone test-zone2: shard=8 master=00000000006.57.17 target=00000000004.52.9 2020-06-04T08:37:33.166 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:33.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:33.169 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:33.223 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:33.223 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:33.468 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.59.14","timestamp":"2020-06-04T08:37:15.173082Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:37:33.468 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-20 2020-06-04T08:37:33.469 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:33.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:33.470 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:33.511 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:33.513 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:33.739 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.59.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:37:33.739 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-20 2020-06-04T08:37:33.740 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:33.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:33.741 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:33.785 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:33.786 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:34.030 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.59.14","timestamp":"2020-06-04T08:37:15.173082Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:37:34.031 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-20 2020-06-04T08:37:34.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:34.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:34.033 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:34.072 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:34.073 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:34.295 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.39.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:37:34.295 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-20 2020-06-04T08:37:34.296 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:34.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:34.297 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:34.381 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:34.382 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:34.638 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.37.17","timestamp":"2020-06-04T08:37:15.173082Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:37:34.639 WARNING:rgw_multi.tests:bucket gpxthc-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000007.39.14 target=00000000006.37.17 2020-06-04T08:37:39.640 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:39.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:39.642 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:39.688 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:39.689 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:39.940 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.37.17","timestamp":"2020-06-04T08:37:15.173082Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:37:39.941 WARNING:rgw_multi.tests:bucket gpxthc-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000007.39.14 target=00000000006.37.17 2020-06-04T08:37:44.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:44.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:44.944 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:44.993 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:44.993 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:45.244 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.37.17","timestamp":"2020-06-04T08:37:15.173082Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:37:45.244 WARNING:rgw_multi.tests:bucket gpxthc-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000007.39.14 target=00000000006.37.17 2020-06-04T08:37:50.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:50.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:50.249 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:50.297 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:50.297 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:50.555 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.37.17","timestamp":"2020-06-04T08:37:15.173082Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:37:50.556 WARNING:rgw_multi.tests:bucket gpxthc-20 zone test-zone1 behind zone test-zone3: shard=8 master=00000000007.39.14 target=00000000006.37.17 2020-06-04T08:37:55.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:55.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:55.560 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:55.606 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:55.607 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:55.853 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.39.14","timestamp":"2020-06-04T08:37:15.173082Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:37:55.854 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-20 2020-06-04T08:37:55.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:55.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:55.856 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:55.941 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:55.941 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:56.164 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.39.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:37:56.165 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-20 2020-06-04T08:37:56.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:56.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:56.167 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:56.208 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:56.208 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:56.449 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.39.14","timestamp":"2020-06-04T08:37:15.173082Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:37:56.450 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-20 2020-06-04T08:37:56.450 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-20 zones={test-zone2, test-zone1} 2020-06-04T08:37:56.465 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:37:56.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:56.520 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:56.521 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:37:56.534 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:56.535 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:56.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:56.617 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:56.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:56.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:56.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:56.861 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:56.934 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-20 zones={test-zone2, test-zone1} 2020-06-04T08:37:56.935 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-20 zones={test-zone3, test-zone1} 2020-06-04T08:37:56.945 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:37:56.998 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:56.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:57.000 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:37:57.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:57.010 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:57.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:57.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:57.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:57.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:57.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:57.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:57.378 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-20 zones={test-zone3, test-zone1} 2020-06-04T08:37:57.379 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-20 zones={test-zone3, test-zone2} 2020-06-04T08:37:57.390 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:37:57.438 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:57.439 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:57.440 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:37:57.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:57.449 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:37:57.513 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:57.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:57.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:57.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:57.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:57.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:37:57.822 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-20 zones={test-zone3, test-zone2} 2020-06-04T08:37:57.823 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2020-06-04T08:37:57.825 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-21 2020-06-04T08:37:57.859 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gpxthc-22 2020-06-04T08:37:57.915 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gpxthc-23 2020-06-04T08:37:58.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:58.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:58.024 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:58.059 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:58.060 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:58.243 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=838ed6b2-fa2f-4917-a5c1-6c82bb405cda 2020-06-04T08:37:58.279 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259878.006196_803.1","last_update":"2020-06-04T08:37:58.006196Z"},{"marker":"1_1591259350.985454_111.1","last_update":"2020-06-04T08:29:10.985454Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:37:58.283 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:37:58.284 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:58.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:58.285 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:58.325 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:58.326 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:58.545 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259878.006196_803.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:37:58.006196Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:37:58.545 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:37:58.546 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:37:58.546 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:58.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:58.547 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:58.587 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:58.588 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:58.813 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259878.006196_803.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:37:58.006196Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:37:58.814 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:37:58.814 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:37:58.815 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:58.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:58.817 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:58.858 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:58.858 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:59.133 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259878.006196_803.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:37:58.006196Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:37:59.134 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:37:59.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:59.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:59.457 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:59.506 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:59.506 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:59.693 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.41.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:37:59.694 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-21 2020-06-04T08:37:59.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:59.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:59.697 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:37:59.742 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:37:59.742 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:37:59.959 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"mdzBt3506PX1IxFfCXzJionyTSSWlRf","ns":""},"count":2},"inc_marker":{"position":"00000000003.41.11","timestamp":"2020-06-04T08:37:59.226083Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:37:59.959 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-21 2020-06-04T08:37:59.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:59.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:37:59.962 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:00.009 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:00.010 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:00.241 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.41.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:00.242 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-21 2020-06-04T08:38:00.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:00.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:00.244 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:00.287 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:00.288 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:00.541 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"mdzBt3506PX1IxFfCXzJionyTSSWlRf","ns":""},"count":1},"inc_marker":{"position":"00000000003.41.11","timestamp":"2020-06-04T08:37:59.226083Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:00.542 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-21 2020-06-04T08:38:00.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:00.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:00.545 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:00.628 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:00.628 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:00.856 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.41.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:00.857 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-21 2020-06-04T08:38:00.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:00.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:00.860 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:00.906 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:00.907 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:01.178 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.41.11","timestamp":"2020-06-04T08:37:59.226083Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:01.179 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-21 2020-06-04T08:38:01.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:01.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:01.181 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:01.221 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:01.222 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:01.449 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.41.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:01.449 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-21 2020-06-04T08:38:01.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:01.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:01.451 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:01.498 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:01.498 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:01.743 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"mdzBt3506PX1IxFfCXzJionyTSSWlRf","ns":""},"count":2},"inc_marker":{"position":"00000000002.41.5","timestamp":"2020-06-04T08:37:59.417494Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:01.744 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-21 2020-06-04T08:38:01.744 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:01.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:01.745 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:01.786 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:01.787 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:02.000 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.41.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:02.001 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-21 2020-06-04T08:38:02.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:02.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:02.003 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:02.051 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:02.051 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:02.302 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"mdzBt3506PX1IxFfCXzJionyTSSWlRf","ns":""},"count":2},"inc_marker":{"position":"00000000002.41.5","timestamp":"2020-06-04T08:37:59.378304Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:02.303 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-21 2020-06-04T08:38:02.303 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:02.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:02.304 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:02.394 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:02.395 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:02.615 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.64.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:02.616 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-21 2020-06-04T08:38:02.617 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:02.619 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:02.620 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:02.702 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:02.703 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:02.947 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"mdzBt3506PX1IxFfCXzJionyTSSWlRf","ns":""},"count":2},"inc_marker":{"position":"00000000003.64.11","timestamp":"2020-06-04T08:37:59.458359Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:02.947 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-21 2020-06-04T08:38:02.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:02.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:02.949 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:03.032 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:03.032 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:03.270 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.64.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:03.271 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-21 2020-06-04T08:38:03.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:03.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:03.274 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:03.317 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:03.318 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:03.560 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"mdzBt3506PX1IxFfCXzJionyTSSWlRf","ns":""},"count":2},"inc_marker":{"position":"00000000003.64.11","timestamp":"2020-06-04T08:37:59.462958Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:03.561 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-21 2020-06-04T08:38:03.562 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-21 zones={test-zone2, test-zone1} 2020-06-04T08:38:03.579 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:38:03.637 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:03.638 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:03.639 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:38:03.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:03.652 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:03.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:03.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:03.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:03.827 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-21 zones={test-zone2, test-zone1} 2020-06-04T08:38:03.828 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-21 zones={test-zone3, test-zone1} 2020-06-04T08:38:03.842 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:38:03.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:03.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:03.897 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:38:03.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:03.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:03.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:04.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:04.021 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:04.102 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-21 zones={test-zone3, test-zone1} 2020-06-04T08:38:04.103 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-21 zones={test-zone3, test-zone2} 2020-06-04T08:38:04.114 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:38:04.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:04.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:04.172 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:38:04.180 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:04.181 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:04.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:04.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:04.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:04.371 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-21 zones={test-zone3, test-zone2} 2020-06-04T08:38:04.372 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:04.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:04.373 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:04.416 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:04.417 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:04.645 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.20.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:04.645 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-22 2020-06-04T08:38:04.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:04.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:04.647 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:04.701 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:04.702 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:04.936 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"yS.g4NV0JOjejggdBddLuYHDqS72frk","ns":""},"count":1},"inc_marker":{"position":"00000000002.20.5","timestamp":"2020-06-04T08:37:59.623439Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:04.937 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-22 2020-06-04T08:38:04.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:04.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:04.939 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:04.982 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:04.982 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:05.217 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.20.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:05.218 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-22 2020-06-04T08:38:05.219 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:05.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:05.220 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:05.304 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:05.304 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:05.547 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"yS.g4NV0JOjejggdBddLuYHDqS72frk","ns":""},"count":1},"inc_marker":{"position":"00000000002.20.5","timestamp":"2020-06-04T08:37:59.617360Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:05.547 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-22 2020-06-04T08:38:05.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:05.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:05.550 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:05.629 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:05.630 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:05.852 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.20.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:05.855 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-22 2020-06-04T08:38:05.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:05.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:05.857 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:05.909 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:05.909 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:06.164 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.20.5","timestamp":"2020-06-04T08:37:59.258771Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:06.165 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-22 2020-06-04T08:38:06.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:06.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:06.167 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:06.253 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:06.254 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:06.478 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.18.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:06.478 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-22 2020-06-04T08:38:06.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:06.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:06.481 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:06.527 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:06.527 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:06.790 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"yS.g4NV0JOjejggdBddLuYHDqS72frk","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.11","timestamp":"2020-06-04T08:37:59.377100Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:06.791 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-22 2020-06-04T08:38:06.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:06.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:06.792 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:06.834 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:06.835 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:07.051 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.18.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:07.051 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-22 2020-06-04T08:38:07.052 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:07.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:07.053 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:07.100 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:07.101 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:07.350 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"yS.g4NV0JOjejggdBddLuYHDqS72frk","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.11","timestamp":"2020-06-04T08:37:59.421454Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:07.351 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-22 2020-06-04T08:38:07.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:07.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:07.354 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:07.432 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:07.433 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:07.663 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.17.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:07.664 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-22 2020-06-04T08:38:07.665 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:07.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:07.666 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:07.758 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:07.758 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:08.002 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"yS.g4NV0JOjejggdBddLuYHDqS72frk","ns":""},"count":1},"inc_marker":{"position":"00000000002.17.5","timestamp":"2020-06-04T08:37:59.701380Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:08.002 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-22 2020-06-04T08:38:08.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:08.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:08.004 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:08.044 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:08.044 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:08.273 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.17.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:08.274 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-22 2020-06-04T08:38:08.275 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:08.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:08.276 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:08.323 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:08.323 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:08.562 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"yS.g4NV0JOjejggdBddLuYHDqS72frk","ns":""},"count":1},"inc_marker":{"position":"00000000002.17.5","timestamp":"2020-06-04T08:37:59.703436Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:08.563 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-22 2020-06-04T08:38:08.564 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-22 zones={test-zone2, test-zone1} 2020-06-04T08:38:08.580 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:38:08.632 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:08.633 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:08.633 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:38:08.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:08.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:08.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:08.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:08.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:08.822 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-22 zones={test-zone2, test-zone1} 2020-06-04T08:38:08.822 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-22 zones={test-zone3, test-zone1} 2020-06-04T08:38:08.841 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:38:08.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:08.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:08.896 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:38:08.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:08.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:08.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:08.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:09.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:09.083 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-22 zones={test-zone3, test-zone1} 2020-06-04T08:38:09.084 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-22 zones={test-zone3, test-zone2} 2020-06-04T08:38:09.092 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:38:09.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:09.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:09.143 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:38:09.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:09.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:09.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:09.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:09.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:09.312 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-22 zones={test-zone3, test-zone2} 2020-06-04T08:38:09.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:09.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:09.314 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:09.356 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:09.356 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:09.588 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.12.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:09.588 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-23 2020-06-04T08:38:09.589 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:09.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:09.590 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:09.632 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:09.632 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:09.884 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"LwZBMG9hsFElRRN7VVrriFOjh-f2cJs","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.10","timestamp":"2020-06-04T08:37:59.581200Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:09.885 WARNING:rgw_multi.tests:bucket gpxthc-23 zone test-zone2 behind zone test-zone1: shard=8 master=00000000002.12.5 target=00000000001.9.10 2020-06-04T08:38:14.887 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:14.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:14.889 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:14.933 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:14.934 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:15.163 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"LwZBMG9hsFElRRN7VVrriFOjh-f2cJs","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.10","timestamp":"2020-06-04T08:37:59.581200Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:15.164 WARNING:rgw_multi.tests:bucket gpxthc-23 zone test-zone2 behind zone test-zone1: shard=8 master=00000000002.12.5 target=00000000001.9.10 2020-06-04T08:38:20.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:20.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:20.167 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:20.214 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:20.214 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:20.454 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"LwZBMG9hsFElRRN7VVrriFOjh-f2cJs","ns":""},"count":1},"inc_marker":{"position":"00000000002.12.5","timestamp":"2020-06-04T08:37:59.374096Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:20.455 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-23 2020-06-04T08:38:20.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:20.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:20.457 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:20.498 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:20.498 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:20.734 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.12.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:20.735 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-23 2020-06-04T08:38:20.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:20.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:20.737 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:20.822 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:20.823 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:21.073 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"LwZBMG9hsFElRRN7VVrriFOjh-f2cJs","ns":""},"count":1},"inc_marker":{"position":"00000000002.12.5","timestamp":"2020-06-04T08:37:59.374096Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:21.073 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-23 2020-06-04T08:38:21.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:21.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:21.075 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:21.116 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:21.117 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:21.365 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.12.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:21.365 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-23 2020-06-04T08:38:21.366 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:21.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:21.367 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:21.413 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:21.414 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:21.683 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.12.5","timestamp":"2020-06-04T08:37:59.374096Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:21.683 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-23 2020-06-04T08:38:21.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:21.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:21.685 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:21.737 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:21.737 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:21.959 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.18.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:21.959 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-23 2020-06-04T08:38:21.960 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:21.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:21.961 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:22.006 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:22.006 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:22.258 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"wx7aELwGuDMKU4xAO.u1hTy98y.kzZe","ns":""},"count":2},"inc_marker":{"position":"00000000002.18.5","timestamp":"2020-06-04T08:37:59.374096Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:22.258 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-23 2020-06-04T08:38:22.259 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:22.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:22.260 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:22.306 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:22.306 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:22.526 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.18.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:22.527 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-23 2020-06-04T08:38:22.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:22.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:22.529 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:22.572 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:22.573 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:22.815 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"wx7aELwGuDMKU4xAO.u1hTy98y.kzZe","ns":""},"count":2},"inc_marker":{"position":"00000000002.18.5","timestamp":"2020-06-04T08:37:59.374096Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:22.816 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-23 2020-06-04T08:38:22.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:22.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:22.817 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:22.898 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:22.898 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:23.138 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.18.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:23.139 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-23 2020-06-04T08:38:23.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:23.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:23.140 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:23.232 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:23.232 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:23.486 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"wx7aELwGuDMKU4xAO.u1hTy98y.kzZe","ns":""},"count":2},"inc_marker":{"position":"00000000003.18.11","timestamp":"2020-06-04T08:37:59.501358Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:23.487 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-23 2020-06-04T08:38:23.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:23.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:23.489 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:23.530 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:23.530 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:23.765 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.18.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:23.766 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-23 2020-06-04T08:38:23.766 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:23.767 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:23.768 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:23.812 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:23.813 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:24.059 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"wx7aELwGuDMKU4xAO.u1hTy98y.kzZe","ns":""},"count":2},"inc_marker":{"position":"00000000003.18.11","timestamp":"2020-06-04T08:37:59.503532Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:24.059 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-23 2020-06-04T08:38:24.060 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-23 zones={test-zone2, test-zone1} 2020-06-04T08:38:24.072 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:38:24.122 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:24.123 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:24.124 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:38:24.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:24.135 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:24.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:24.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:24.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:24.301 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-23 zones={test-zone2, test-zone1} 2020-06-04T08:38:24.302 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-23 zones={test-zone3, test-zone1} 2020-06-04T08:38:24.314 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:38:24.362 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:24.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:24.364 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:38:24.374 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:24.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:24.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:24.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:24.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:24.538 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-23 zones={test-zone3, test-zone1} 2020-06-04T08:38:24.539 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-23 zones={test-zone3, test-zone2} 2020-06-04T08:38:24.549 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:38:24.599 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:24.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:24.601 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:38:24.609 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:24.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:24.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:24.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:24.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:24.794 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-23 zones={test-zone3, test-zone2} 2020-06-04T08:38:24.795 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_delete_marker_full_sync ... ok 2020-06-04T08:38:24.796 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-24 2020-06-04T08:38:24.825 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gpxthc-25 2020-06-04T08:38:24.888 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gpxthc-26 2020-06-04T08:38:25.076 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:25.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:25.077 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:25.130 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:25.130 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:25.329 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=838ed6b2-fa2f-4917-a5c1-6c82bb405cda 2020-06-04T08:38:25.367 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259905.064571_843.1","last_update":"2020-06-04T08:38:25.064571Z"},{"marker":"1_1591259350.985454_111.1","last_update":"2020-06-04T08:29:10.985454Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:38:25.368 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:38:25.368 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:25.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:25.370 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:25.421 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:25.421 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:25.646 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259905.023462_839.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:38:25.023462Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:38:25.647 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259905.064571_843.1 target=1_1591259905.023462_839.1 2020-06-04T08:38:30.649 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:30.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:30.652 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:30.694 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:30.694 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:30.916 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259905.064571_843.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:38:25.064571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:38:30.917 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:38:30.918 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:38:30.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:30.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:30.919 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:30.962 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:30.962 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:31.206 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259905.064571_843.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:38:25.064571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:38:31.207 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:38:31.208 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:38:31.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:31.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:31.209 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:31.248 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:31.248 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:31.517 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259905.064571_843.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:38:25.064571Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:38:31.517 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:38:31.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:31.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:31.772 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:31.816 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:31.816 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:32.014 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.25.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:32.015 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-24 2020-06-04T08:38:32.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:32.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:32.016 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:32.066 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:32.067 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:32.274 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.25.6","timestamp":"2020-06-04T08:38:31.643440Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:32.275 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-24 2020-06-04T08:38:32.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:32.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:32.276 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:32.319 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:32.319 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:32.553 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.25.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:32.554 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-24 2020-06-04T08:38:32.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:32.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:32.556 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:32.639 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:32.640 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:32.884 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.25.6","timestamp":"2020-06-04T08:38:31.637548Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:32.884 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-24 2020-06-04T08:38:32.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:32.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:32.886 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:32.965 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:32.965 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:33.203 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.25.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:33.204 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-24 2020-06-04T08:38:33.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:33.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:33.206 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:33.249 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:33.249 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:33.516 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.25.6","timestamp":"2020-06-04T08:38:31.526179Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:33.517 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-24 2020-06-04T08:38:33.518 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:33.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:33.519 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:33.600 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:33.601 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:33.819 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.23.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:33.820 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-24 2020-06-04T08:38:33.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:33.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:33.822 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:33.867 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:33.868 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:34.116 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.23.12","timestamp":"2020-06-04T08:38:31.841499Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:34.117 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-24 2020-06-04T08:38:34.117 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:34.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:34.118 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:34.161 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:34.161 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:34.391 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.23.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:34.391 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-24 2020-06-04T08:38:34.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:34.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:34.393 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:34.439 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:34.440 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:34.699 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.23.12","timestamp":"2020-06-04T08:38:31.798993Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:34.700 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-24 2020-06-04T08:38:34.700 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:34.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:34.702 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:34.783 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:34.783 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:35.024 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.23.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:35.024 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-24 2020-06-04T08:38:35.025 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:35.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:35.026 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:35.110 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:35.110 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:35.363 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.23.12","timestamp":"2020-06-04T08:38:31.877443Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:35.363 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-24 2020-06-04T08:38:35.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:35.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:35.365 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:35.407 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:35.408 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:35.642 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.23.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:35.642 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-24 2020-06-04T08:38:35.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:35.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:35.644 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:35.686 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:35.687 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:35.930 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.23.12","timestamp":"2020-06-04T08:38:31.923452Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:35.930 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-24 2020-06-04T08:38:35.931 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-24 zones={test-zone2, test-zone1} 2020-06-04T08:38:35.944 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:38:35.999 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:36.000 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:38:36.010 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:36.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:36.084 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-24 zones={test-zone2, test-zone1} 2020-06-04T08:38:36.085 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-24 zones={test-zone3, test-zone1} 2020-06-04T08:38:36.099 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:38:36.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:36.151 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:38:36.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:36.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:36.229 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-24 zones={test-zone3, test-zone1} 2020-06-04T08:38:36.230 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-24 zones={test-zone3, test-zone2} 2020-06-04T08:38:36.241 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:38:36.290 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:36.291 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:38:36.299 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:36.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:36.368 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-24 zones={test-zone3, test-zone2} 2020-06-04T08:38:36.369 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:36.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:36.371 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:36.413 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:36.413 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:36.640 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.41.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:36.641 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-25 2020-06-04T08:38:36.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:36.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:36.643 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:36.694 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:36.694 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:36.926 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.12","timestamp":"2020-06-04T08:38:32.047476Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:36.927 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-25 2020-06-04T08:38:36.927 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:36.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:36.928 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:36.972 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:36.973 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:37.207 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.41.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:37.208 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-25 2020-06-04T08:38:37.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:37.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:37.210 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:37.292 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:37.292 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:37.539 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.12","timestamp":"2020-06-04T08:38:32.041493Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:37.539 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-25 2020-06-04T08:38:37.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:37.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:37.542 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:37.623 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:37.623 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:37.858 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.41.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:37.859 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-25 2020-06-04T08:38:37.860 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:37.860 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:37.861 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:37.907 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:37.907 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:38.182 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.41.12","timestamp":"2020-06-04T08:38:31.666816Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:38.182 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-25 2020-06-04T08:38:38.183 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:38.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:38.185 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:38.273 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:38.274 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:38.497 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.40.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:38.498 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-25 2020-06-04T08:38:38.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:38.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:38.500 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:38.543 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:38.544 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:38.802 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.40.6","timestamp":"2020-06-04T08:38:31.800608Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:38.803 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-25 2020-06-04T08:38:38.804 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:38.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:38.805 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:38.847 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:38.848 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:39.076 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.40.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:39.077 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-25 2020-06-04T08:38:39.077 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:39.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:39.079 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:39.125 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:39.125 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:39.364 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.40.6","timestamp":"2020-06-04T08:38:31.841496Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:39.365 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-25 2020-06-04T08:38:39.366 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:39.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:39.367 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:39.447 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:39.448 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:39.678 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.33.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:39.678 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-25 2020-06-04T08:38:39.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:39.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:39.680 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:39.768 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:39.768 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:40.011 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.33.12","timestamp":"2020-06-04T08:38:32.161494Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:40.012 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-25 2020-06-04T08:38:40.012 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:40.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:40.013 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:40.054 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:40.055 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:40.293 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.33.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:40.293 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-25 2020-06-04T08:38:40.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:40.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:40.295 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:40.341 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:40.341 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:40.580 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.33.12","timestamp":"2020-06-04T08:38:32.163429Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:40.580 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-25 2020-06-04T08:38:40.581 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-25 zones={test-zone2, test-zone1} 2020-06-04T08:38:40.595 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:38:40.642 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:40.643 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:38:40.654 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:40.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:40.725 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-25 zones={test-zone2, test-zone1} 2020-06-04T08:38:40.726 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-25 zones={test-zone3, test-zone1} 2020-06-04T08:38:40.735 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:38:40.786 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:40.787 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:38:40.794 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:40.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:40.855 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-25 zones={test-zone3, test-zone1} 2020-06-04T08:38:40.855 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-25 zones={test-zone3, test-zone2} 2020-06-04T08:38:40.866 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:38:40.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:40.915 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:38:40.938 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:40.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:41.010 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-25 zones={test-zone3, test-zone2} 2020-06-04T08:38:41.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:41.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:41.013 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:41.056 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:41.056 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:41.295 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.17.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:41.296 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-26 2020-06-04T08:38:41.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:41.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:41.297 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:41.342 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:41.343 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:41.577 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.17.12","timestamp":"2020-06-04T08:38:32.001302Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:41.577 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-26 2020-06-04T08:38:41.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:41.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:41.579 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:41.620 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:41.620 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:41.839 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.17.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:41.839 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-26 2020-06-04T08:38:41.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:41.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:41.841 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:41.926 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:41.926 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:42.188 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.17.12","timestamp":"2020-06-04T08:38:31.994570Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:42.189 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-26 2020-06-04T08:38:42.190 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:42.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:42.191 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:42.271 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:42.272 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:42.519 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.17.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:42.520 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-26 2020-06-04T08:38:42.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:42.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:42.522 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:42.565 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:42.566 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:42.841 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.17.12","timestamp":"2020-06-04T08:38:31.742292Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:42.842 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-26 2020-06-04T08:38:42.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:42.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:42.843 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:42.922 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:42.923 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:43.135 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.13.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:43.135 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-26 2020-06-04T08:38:43.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:43.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:43.137 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:43.181 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:43.181 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:43.435 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.12","timestamp":"2020-06-04T08:38:32.037505Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:43.436 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-26 2020-06-04T08:38:43.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:43.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:43.437 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:43.479 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:43.480 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:43.703 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.13.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:43.703 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-26 2020-06-04T08:38:43.704 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:43.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:43.705 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:43.755 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:43.756 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:44.012 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.12","timestamp":"2020-06-04T08:38:32.085434Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:44.012 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-26 2020-06-04T08:38:44.013 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:44.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:44.014 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:44.096 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:44.097 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:44.328 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.22.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:44.329 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-26 2020-06-04T08:38:44.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:44.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:44.331 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:44.412 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:44.412 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:44.672 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.22.6","timestamp":"2020-06-04T08:38:31.917588Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:44.672 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-26 2020-06-04T08:38:44.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:44.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:44.674 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:44.755 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:44.755 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:44.987 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.22.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:38:44.988 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-26 2020-06-04T08:38:44.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:44.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:44.990 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:45.034 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:45.035 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:45.274 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.22.6","timestamp":"2020-06-04T08:38:31.882380Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:38:45.274 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-26 2020-06-04T08:38:45.275 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-26 zones={test-zone2, test-zone1} 2020-06-04T08:38:45.286 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:38:45.334 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:45.335 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:38:45.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:45.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:45.416 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-26 zones={test-zone2, test-zone1} 2020-06-04T08:38:45.417 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-26 zones={test-zone3, test-zone1} 2020-06-04T08:38:45.428 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:38:45.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:45.479 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:38:45.488 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:45.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:45.557 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-26 zones={test-zone3, test-zone1} 2020-06-04T08:38:45.557 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-26 zones={test-zone3, test-zone2} 2020-06-04T08:38:45.580 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:38:45.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:45.631 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:38:45.639 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2020-06-04T08:38:45.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2020-06-04T08:38:45.705 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-26 zones={test-zone3, test-zone2} 2020-06-04T08:38:45.706 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2020-06-04T08:38:45.707 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-27 2020-06-04T08:38:45.745 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gpxthc-28 2020-06-04T08:38:45.809 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gpxthc-29 2020-06-04T08:38:46.062 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_versioning ... ok 2020-06-04T08:38:46.063 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-30 2020-06-04T08:38:46.097 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gpxthc-31 2020-06-04T08:38:46.157 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gpxthc-32 2020-06-04T08:38:46.574 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_acl ... ok 2020-06-04T08:38:46.576 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-33 2020-06-04T08:38:46.624 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gpxthc-34 2020-06-04T08:38:46.676 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gpxthc-35 2020-06-04T08:38:48.934 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_cors ... ok 2020-06-04T08:38:48.935 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-36 2020-06-04T08:38:48.974 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gpxthc-37 2020-06-04T08:38:49.034 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gpxthc-38 2020-06-04T08:38:49.086 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:49.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:49.088 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:49.123 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:49.124 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:49.307 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=838ed6b2-fa2f-4917-a5c1-6c82bb405cda 2020-06-04T08:38:49.345 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259929.063266_929.1","last_update":"2020-06-04T08:38:49.063266Z"},{"marker":"1_1591259350.985454_111.1","last_update":"2020-06-04T08:29:10.985454Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:38:49.346 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:38:49.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:49.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:49.348 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:49.391 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:49.392 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:49.618 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259929.063266_929.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:38:49.063266Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:38:49.620 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:38:49.620 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:38:49.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:49.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:49.622 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:49.668 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:49.669 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:49.886 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259929.063266_929.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:38:49.063266Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:38:49.887 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:38:49.887 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:38:49.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:49.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:49.889 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:49.931 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:49.932 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:50.209 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259929.063266_929.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:38:49.063266Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:38:50.209 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:38:50.600 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_delete_notempty ... ok 2020-06-04T08:38:50.601 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-39 2020-06-04T08:38:50.627 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gpxthc-40 2020-06-04T08:38:50.690 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gpxthc-41 2020-06-04T08:38:50.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:50.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:50.754 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:50.796 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:50.797 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:50.971 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=838ed6b2-fa2f-4917-a5c1-6c82bb405cda 2020-06-04T08:38:51.004 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259930.725192_942.1","last_update":"2020-06-04T08:38:50.725192Z"},{"marker":"1_1591259350.985454_111.1","last_update":"2020-06-04T08:29:10.985454Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:38:51.005 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:38:51.006 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:51.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:51.008 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:51.050 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:51.050 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:51.261 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259930.719322_940.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:38:50.719322Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:38:51.262 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591259930.725192_942.1 target=1_1591259930.719322_940.1 2020-06-04T08:38:56.264 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:56.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:56.266 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:56.307 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:56.308 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:56.527 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259930.725192_942.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:38:50.725192Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:38:56.528 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:38:56.529 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:38:56.529 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:56.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:56.530 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:56.572 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:56.572 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:56.800 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259930.725192_942.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:38:50.725192Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:38:56.800 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:38:56.801 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:38:56.801 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:56.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:38:56.802 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:38:56.842 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:38:56.842 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:38:57.107 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591259930.725192_942.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:38:50.725192Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:38:57.108 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:38:57.109 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2020-06-04T08:38:57.109 INFO:teuthology.orchestra.run:waiting for 300 2020-06-04T08:38:57.110 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-06-04T08:38:57.103+0000 7fe1577fe700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 18593) UID: 0 2020-06-04T08:38:57.110 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-06-04T08:38:57.103+0000 7fe1731780c0 -1 shutting down 2020-06-04T08:39:01.005 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2020-06-04T08:39:03.212 INFO:tasks.rgw.c1.client.1:Stopped 2020-06-04T08:39:03.212 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:39:03.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:39:03.214 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:39:03.252 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:39:03.252 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:39:03.285 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:39:03.275+0000 7f0ec91b00c0 0 NOTICE: overriding master zone: 7ad4cc51-2d19-48f1-a21e-ef15faf62b94 2020-06-04T08:39:03.286 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:39:03.279+0000 7f0ec91b00c0 0 failed reading obj info from .rgw.root:zone_info.463a7035-f6f9-4bfc-a894-c421122fd949: (2) No such file or directory 2020-06-04T08:39:03.286 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:39:03.279+0000 7f0ec91b00c0 0 WARNING: could not read zone params for zone id=463a7035-f6f9-4bfc-a894-c421122fd949 name=test-zone3 2020-06-04T08:39:03.286 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:39:03.279+0000 7f0ec91b00c0 0 failed reading obj info from .rgw.root:zone_info.7ad4cc51-2d19-48f1-a21e-ef15faf62b94: (2) No such file or directory 2020-06-04T08:39:03.286 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:39:03.279+0000 7f0ec91b00c0 0 WARNING: could not read zone params for zone id=7ad4cc51-2d19-48f1-a21e-ef15faf62b94 name=test-zone1 2020-06-04T08:39:03.301 INFO:teuthology.orchestra.run.smithi196.stdout:{"id":"d7df857b-1e19-4b4c-aa22-43c42d072194","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"M2KEYTP439P9KXXD","secret_key":"Ad56bMSsbqPkjfMTVush7dBgYdLf27ul"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","notif_pool":"test-zone2.rgw.log:notif"} 2020-06-04T08:39:03.302 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:39:03.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:39:03.304 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:39:03.383 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:39:03.383 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:39:03.678 INFO:teuthology.orchestra.run.smithi196.stdout:{"id":"8681f463-3379-4ae0-aef4-6adb4b0f951f","epoch":1,"predecessor_uuid":"838ed6b2-fa2f-4917-a5c1-6c82bb405cda","sync_status":["1_1591259930.725192_942.1","1_1591259350.985454_111.1","",""],"period_map":{"id":"8681f463-3379-4ae0-aef4-6adb4b0f951f","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"d7df857b-1e19-4b4c-aa22-43c42d072194","zones":[{"id":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","name":"test-zone4","endpoints":["http://smithi196.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"463a7035-f6f9-4bfc-a894-c421122fd949","name":"test-zone3","endpoints":["http://smithi060.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","name":"test-zone1","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7df857b-1e19-4b4c-aa22-43c42d072194","name":"test-zone2","endpoints":["http://smithi196.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","val":2528435237},{"key":"463a7035-f6f9-4bfc-a894-c421122fd949","val":2270195151},{"key":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","val":3611466229},{"key":"d7df857b-1e19-4b4c-aa22-43c42d072194","val":443345077}]},"master_zonegroup":"test-zonegroup","master_zone":"d7df857b-1e19-4b4c-aa22-43c42d072194","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","realm_name":"test-realm","realm_epoch":3} 2020-06-04T08:39:03.679 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2020-06-04T08:39:06.608 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2020-06-04T08:40:03.681 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-42 2020-06-04T08:40:03.760 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gpxthc-43 2020-06-04T08:40:03.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:40:03.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:40:03.800 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:40:03.837 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:40:03.837 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:40:04.025 INFO:teuthology.orchestra.run.smithi196.stderr:No --period given, using current period=8681f463-3379-4ae0-aef4-6adb4b0f951f 2020-06-04T08:40:04.047 INFO:teuthology.orchestra.run.smithi196.stdout:[{"marker":"1_1591260003.794286_107.1","last_update":"2020-06-04T08:40:03.794286Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:40:04.048 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2020-06-04T08:40:04.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:40:04.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:40:04.051 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:40:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:40:04.095 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:40:04.332 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"8681f463-3379-4ae0-aef4-6adb4b0f951f","realm_epoch":3},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260003.794286_107.1","next_step_marker":"","total_entries":81,"pos":0,"timestamp":"2020-06-04T08:40:03.794286Z","realm_epoch":3}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}}]},"full_sync":{"total":81,"complete":81}} 2020-06-04T08:40:04.333 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2020-06-04T08:40:04.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:40:04.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:40:04.335 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:40:04.415 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:40:04.415 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:40:04.444 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:40:04.434+0000 7f4a12c200c0 0 NOTICE: overriding master zone: d7df857b-1e19-4b4c-aa22-43c42d072194 2020-06-04T08:40:04.444 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:40:04.434+0000 7f4a12c200c0 0 failed reading obj info from .rgw.root:zone_info.38aeb3a1-6a1a-479f-a31a-ae49981e3118: (2) No such file or directory 2020-06-04T08:40:04.445 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:40:04.434+0000 7f4a12c200c0 0 WARNING: could not read zone params for zone id=38aeb3a1-6a1a-479f-a31a-ae49981e3118 name=test-zone4 2020-06-04T08:40:04.446 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:40:04.438+0000 7f4a12c200c0 0 failed reading obj info from .rgw.root:zone_info.d7df857b-1e19-4b4c-aa22-43c42d072194: (2) No such file or directory 2020-06-04T08:40:04.447 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:40:04.438+0000 7f4a12c200c0 0 WARNING: could not read zone params for zone id=d7df857b-1e19-4b4c-aa22-43c42d072194 name=test-zone2 2020-06-04T08:40:04.459 INFO:teuthology.orchestra.run.smithi060.stdout:{"id":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"M2KEYTP439P9KXXD","secret_key":"Ad56bMSsbqPkjfMTVush7dBgYdLf27ul"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","notif_pool":"test-zone1.rgw.log:notif"} 2020-06-04T08:40:04.462 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:40:04.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:40:04.463 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:40:04.547 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:40:04.548 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:40:04.841 INFO:teuthology.orchestra.run.smithi060.stdout:{"id":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","epoch":1,"predecessor_uuid":"8681f463-3379-4ae0-aef4-6adb4b0f951f","sync_status":["1_1591260003.794286_107.1","","",""],"period_map":{"id":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","zones":[{"id":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","name":"test-zone4","endpoints":["http://smithi196.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"463a7035-f6f9-4bfc-a894-c421122fd949","name":"test-zone3","endpoints":["http://smithi060.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","name":"test-zone1","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7df857b-1e19-4b4c-aa22-43c42d072194","name":"test-zone2","endpoints":["http://smithi196.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","val":2528435237},{"key":"463a7035-f6f9-4bfc-a894-c421122fd949","val":2270195151},{"key":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","val":3611466229},{"key":"d7df857b-1e19-4b4c-aa22-43c42d072194","val":443345077}]},"master_zonegroup":"test-zonegroup","master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","realm_name":"test-realm","realm_epoch":4} 2020-06-04T08:40:04.841 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2020-06-04T08:41:04.843 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-44 2020-06-04T08:41:04.893 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gpxthc-45 2020-06-04T08:41:04.966 INFO:tasks.rgw.c1.client.1:Restarting daemon 2020-06-04T08:41:04.967 INFO:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2020-06-04T08:41:04.971 INFO:tasks.rgw.c1.client.1:Started 2020-06-04T08:41:04.971 INFO:teuthology.orchestra.run.smithi060:> curl http://smithi060.front.sepia.ceph.com:8001 2020-06-04T08:41:04.987 INFO:teuthology.orchestra.run.smithi060.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-04T08:41:04.987 INFO:teuthology.orchestra.run.smithi060.stderr: Dload Upload Total Spent Left Speed 2020-06-04T08:41:04.991 INFO:teuthology.orchestra.run.smithi060.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi060.front.sepia.ceph.com port 8001: Connection refused 2020-06-04T08:41:04.992 DEBUG:teuthology.orchestra.run:got remote process result: 7 2020-06-04T08:41:05.993 INFO:teuthology.orchestra.run.smithi060:> curl http://smithi060.front.sepia.ceph.com:8001 2020-06-04T08:41:06.011 INFO:teuthology.orchestra.run.smithi060.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2020-06-04T08:41:06.011 INFO:teuthology.orchestra.run.smithi060.stderr: Dload Upload Total Spent Left Speed 2020-06-04T08:41:06.011 INFO:teuthology.orchestra.run.smithi060.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2020-06-04T08:41:06.012 INFO:teuthology.orchestra.run.smithi060.stdout:anonymous 2020-06-04T08:41:06.013 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:06.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:06.014 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:06.094 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:06.095 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:06.293 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=387a2dc2-99d4-41c4-ad70-50eff9e4eab1 2020-06-04T08:41:06.328 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260064.937745_93.1","last_update":"2020-06-04T08:41:04.937745Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:41:06.328 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:41:06.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:06.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:06.330 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:06.368 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:06.369 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:06.589 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260064.937745_93.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:41:04.937745Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:41:06.590 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:41:06.590 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:41:06.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:06.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:06.592 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:06.632 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:06.632 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:06.842 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260064.937745_93.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:41:04.937745Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:41:06.843 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:41:06.843 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:41:06.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:06.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:06.845 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:06.886 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:06.887 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:07.158 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260064.937745_93.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:41:04.937745Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:41:07.159 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:41:07.159 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-39 zones={test-zone2, test-zone1} 2020-06-04T08:41:07.178 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:07.227 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:07.301 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-39 zones={test-zone2, test-zone1} 2020-06-04T08:41:07.301 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-39 zones={test-zone3, test-zone1} 2020-06-04T08:41:07.320 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:07.374 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:07.443 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-39 zones={test-zone3, test-zone1} 2020-06-04T08:41:07.444 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-39 zones={test-zone1, test-zone2} 2020-06-04T08:41:07.455 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:07.502 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:07.574 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-39 zones={test-zone1, test-zone2} 2020-06-04T08:41:07.575 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-39 zones={test-zone3, test-zone2} 2020-06-04T08:41:07.589 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:07.642 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:07.709 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-39 zones={test-zone3, test-zone2} 2020-06-04T08:41:07.710 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-39 zones={test-zone1, test-zone3} 2020-06-04T08:41:07.721 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:07.771 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:07.845 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-39 zones={test-zone1, test-zone3} 2020-06-04T08:41:07.846 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-39 zones={test-zone2, test-zone3} 2020-06-04T08:41:07.857 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:07.906 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:07.971 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-39 zones={test-zone2, test-zone3} 2020-06-04T08:41:07.972 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-40 zones={test-zone2, test-zone1} 2020-06-04T08:41:07.986 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:08.034 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:08.105 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-40 zones={test-zone2, test-zone1} 2020-06-04T08:41:08.106 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-40 zones={test-zone3, test-zone1} 2020-06-04T08:41:08.117 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:08.170 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:08.243 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-40 zones={test-zone3, test-zone1} 2020-06-04T08:41:08.243 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-40 zones={test-zone1, test-zone2} 2020-06-04T08:41:08.256 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:08.306 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:08.378 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-40 zones={test-zone1, test-zone2} 2020-06-04T08:41:08.378 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-40 zones={test-zone3, test-zone2} 2020-06-04T08:41:08.389 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:08.439 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:08.505 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-40 zones={test-zone3, test-zone2} 2020-06-04T08:41:08.505 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-40 zones={test-zone1, test-zone3} 2020-06-04T08:41:08.519 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:08.566 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:08.634 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-40 zones={test-zone1, test-zone3} 2020-06-04T08:41:08.635 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-40 zones={test-zone2, test-zone3} 2020-06-04T08:41:08.651 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:08.702 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:08.774 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-40 zones={test-zone2, test-zone3} 2020-06-04T08:41:08.775 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-41 zones={test-zone2, test-zone1} 2020-06-04T08:41:08.789 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:08.839 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:08.907 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-41 zones={test-zone2, test-zone1} 2020-06-04T08:41:08.908 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-41 zones={test-zone3, test-zone1} 2020-06-04T08:41:08.920 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:08.974 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:09.042 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-41 zones={test-zone3, test-zone1} 2020-06-04T08:41:09.043 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-41 zones={test-zone1, test-zone2} 2020-06-04T08:41:09.056 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:09.110 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:09.179 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-41 zones={test-zone1, test-zone2} 2020-06-04T08:41:09.180 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-41 zones={test-zone3, test-zone2} 2020-06-04T08:41:09.191 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:09.242 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:09.308 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-41 zones={test-zone3, test-zone2} 2020-06-04T08:41:09.308 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-41 zones={test-zone1, test-zone3} 2020-06-04T08:41:09.322 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:09.370 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:09.442 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-41 zones={test-zone1, test-zone3} 2020-06-04T08:41:09.443 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-41 zones={test-zone2, test-zone3} 2020-06-04T08:41:09.457 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:09.509 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:09.579 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-41 zones={test-zone2, test-zone3} 2020-06-04T08:41:09.580 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-42 zones={test-zone2, test-zone1} 2020-06-04T08:41:09.594 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:09.642 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:09.715 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-42 zones={test-zone2, test-zone1} 2020-06-04T08:41:09.716 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-42 zones={test-zone3, test-zone1} 2020-06-04T08:41:09.725 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:09.774 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:09.848 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-42 zones={test-zone3, test-zone1} 2020-06-04T08:41:09.849 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-42 zones={test-zone1, test-zone2} 2020-06-04T08:41:09.867 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:09.918 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:09.990 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-42 zones={test-zone1, test-zone2} 2020-06-04T08:41:09.991 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-42 zones={test-zone3, test-zone2} 2020-06-04T08:41:10.008 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:10.060 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:10.129 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-42 zones={test-zone3, test-zone2} 2020-06-04T08:41:10.130 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-42 zones={test-zone1, test-zone3} 2020-06-04T08:41:10.142 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:10.190 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:10.260 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-42 zones={test-zone1, test-zone3} 2020-06-04T08:41:10.260 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-42 zones={test-zone2, test-zone3} 2020-06-04T08:41:10.271 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:10.318 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:10.391 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-42 zones={test-zone2, test-zone3} 2020-06-04T08:41:10.392 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-43 zones={test-zone2, test-zone1} 2020-06-04T08:41:10.402 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:10.450 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:10.520 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-43 zones={test-zone2, test-zone1} 2020-06-04T08:41:10.521 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-43 zones={test-zone3, test-zone1} 2020-06-04T08:41:10.529 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:10.578 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:10.655 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-43 zones={test-zone3, test-zone1} 2020-06-04T08:41:10.655 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-43 zones={test-zone1, test-zone2} 2020-06-04T08:41:10.671 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:10.722 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:10.789 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-43 zones={test-zone1, test-zone2} 2020-06-04T08:41:10.790 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-43 zones={test-zone3, test-zone2} 2020-06-04T08:41:10.801 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:10.850 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:10.917 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-43 zones={test-zone3, test-zone2} 2020-06-04T08:41:10.917 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-43 zones={test-zone1, test-zone3} 2020-06-04T08:41:10.930 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:10.982 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:11.050 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-43 zones={test-zone1, test-zone3} 2020-06-04T08:41:11.051 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-43 zones={test-zone2, test-zone3} 2020-06-04T08:41:11.064 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:11.114 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:11.192 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-43 zones={test-zone2, test-zone3} 2020-06-04T08:41:11.193 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-44 zones={test-zone2, test-zone1} 2020-06-04T08:41:11.205 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:11.254 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:11.329 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-44 zones={test-zone2, test-zone1} 2020-06-04T08:41:11.330 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-44 zones={test-zone3, test-zone1} 2020-06-04T08:41:11.339 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:11.390 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:11.464 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-44 zones={test-zone3, test-zone1} 2020-06-04T08:41:11.464 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-44 zones={test-zone1, test-zone2} 2020-06-04T08:41:11.478 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:11.526 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:11.611 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-44 zones={test-zone1, test-zone2} 2020-06-04T08:41:11.611 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-44 zones={test-zone3, test-zone2} 2020-06-04T08:41:11.627 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:11.678 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:11.746 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-44 zones={test-zone3, test-zone2} 2020-06-04T08:41:11.747 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-44 zones={test-zone1, test-zone3} 2020-06-04T08:41:11.760 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:11.806 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:11.882 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-44 zones={test-zone1, test-zone3} 2020-06-04T08:41:11.883 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-44 zones={test-zone2, test-zone3} 2020-06-04T08:41:11.895 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:11.942 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:12.015 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-44 zones={test-zone2, test-zone3} 2020-06-04T08:41:12.016 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-45 zones={test-zone2, test-zone1} 2020-06-04T08:41:12.030 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:12.082 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:12.152 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-45 zones={test-zone2, test-zone1} 2020-06-04T08:41:12.153 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-45 zones={test-zone3, test-zone1} 2020-06-04T08:41:12.162 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:12.210 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:12.284 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-45 zones={test-zone3, test-zone1} 2020-06-04T08:41:12.285 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-45 zones={test-zone1, test-zone2} 2020-06-04T08:41:12.296 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:12.346 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:12.413 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-45 zones={test-zone1, test-zone2} 2020-06-04T08:41:12.414 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-45 zones={test-zone3, test-zone2} 2020-06-04T08:41:12.426 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:12.478 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:12.546 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-45 zones={test-zone3, test-zone2} 2020-06-04T08:41:12.547 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-45 zones={test-zone1, test-zone3} 2020-06-04T08:41:12.562 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:12.610 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:12.675 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-45 zones={test-zone1, test-zone3} 2020-06-04T08:41:12.676 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-45 zones={test-zone2, test-zone3} 2020-06-04T08:41:12.687 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:41:12.739 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:41:12.813 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-45 zones={test-zone2, test-zone3} 2020-06-04T08:41:12.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '838ed6b2-fa2f-4917-a5c1-6c82bb405cda', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:12.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '838ed6b2-fa2f-4917-a5c1-6c82bb405cda', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:12.815 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 838ed6b2-fa2f-4917-a5c1-6c82bb405cda --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:12.851 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:12.851 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:13.148 INFO:teuthology.orchestra.run.smithi060.stdout:[{"id":"1_1591259219.822557_39.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-06-04T08:26:59.822557133Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UKgIH850zXzbqniMBbCDHLr","ver":1},"status":{"status":"write"}}},{"id":"1_1591259219.830770_41.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-06-04T08:26:59.830770292Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UKgIH850zXzbqniMBbCDHLr","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.183102_42.1","section":"bucket.instance","name":"gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19","timestamp":"2020-06-04T08:29:12.183102477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VfH48PXmsZggFiAsqBANxvt","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.187271_50.1","section":"bucket.instance","name":"gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19","timestamp":"2020-06-04T08:29:12.187271885Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VfH48PXmsZggFiAsqBANxvt","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.189031_51.1","section":"bucket","name":"gpxthc-10","timestamp":"2020-06-04T08:29:12.189031257Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_h8WUqJo-QErjBMZIQdgNCtn","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.192886_52.1","section":"bucket","name":"gpxthc-10","timestamp":"2020-06-04T08:29:12.192886901Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_h8WUqJo-QErjBMZIQdgNCtn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.214300_53.1","section":"bucket.instance","name":"gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20","timestamp":"2020-06-04T08:29:12.214300618Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OITM9jIqMd3jh6APsdWU31A","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.219348_54.1","section":"bucket.instance","name":"gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20","timestamp":"2020-06-04T08:29:12.219348485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OITM9jIqMd3jh6APsdWU31A","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.221278_55.1","section":"bucket","name":"gpxthc-11","timestamp":"2020-06-04T08:29:12.221278412Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-nUpFbAACCD5rYXC7DcxUQR","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.225194_56.1","section":"bucket","name":"gpxthc-11","timestamp":"2020-06-04T08:29:12.225194355Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-nUpFbAACCD5rYXC7DcxUQR","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.268612_57.1","section":"bucket.instance","name":"gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21","timestamp":"2020-06-04T08:29:12.268612619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zd8ii08n-ZihNb68aNArIKM","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.273166_58.1","section":"bucket.instance","name":"gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21","timestamp":"2020-06-04T08:29:12.273166922Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zd8ii08n-ZihNb68aNArIKM","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.275116_59.1","section":"bucket","name":"gpxthc-12","timestamp":"2020-06-04T08:29:12.275116572Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AleC4XiO8GV6Mta70FMehuN","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.279189_60.1","section":"bucket","name":"gpxthc-12","timestamp":"2020-06-04T08:29:12.279189055Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AleC4XiO8GV6Mta70FMehuN","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.619667_61.1","section":"bucket.instance","name":"gpxthc-13:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.22","timestamp":"2020-06-04T08:29:31.619667228Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G4zU5SP7vzOavY4UDRy0kVT","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.624339_68.1","section":"bucket.instance","name":"gpxthc-13:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.22","timestamp":"2020-06-04T08:29:31.624339358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G4zU5SP7vzOavY4UDRy0kVT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.626643_69.1","section":"bucket","name":"gpxthc-13","timestamp":"2020-06-04T08:29:31.626643743Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VUOAgr-b5NnlCfY5DmWhaqq","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.630935_70.1","section":"bucket","name":"gpxthc-13","timestamp":"2020-06-04T08:29:31.630935737Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VUOAgr-b5NnlCfY5DmWhaqq","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.653316_71.1","section":"bucket.instance","name":"gpxthc-14:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.23","timestamp":"2020-06-04T08:29:31.653316611Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YhqpqT7kUaddbo1-1KfMhvv","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.658400_72.1","section":"bucket.instance","name":"gpxthc-14:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.23","timestamp":"2020-06-04T08:29:31.658400301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YhqpqT7kUaddbo1-1KfMhvv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.660536_73.1","section":"bucket","name":"gpxthc-14","timestamp":"2020-06-04T08:29:31.660536384Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Br3NXT9kNBdoMbsZqQdcMrv","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.664400_74.1","section":"bucket","name":"gpxthc-14","timestamp":"2020-06-04T08:29:31.664400106Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Br3NXT9kNBdoMbsZqQdcMrv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.712373_75.1","section":"bucket.instance","name":"gpxthc-15:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.24","timestamp":"2020-06-04T08:29:31.712373958Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_frk3BKkQ_Qa4KdotA_oTnm0","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.716427_76.1","section":"bucket.instance","name":"gpxthc-15:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.24","timestamp":"2020-06-04T08:29:31.716427746Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_frk3BKkQ_Qa4KdotA_oTnm0","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.719782_77.1","section":"bucket","name":"gpxthc-15","timestamp":"2020-06-04T08:29:31.719782440Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CRGlFFvUIxHRKMSeAVvS4kF","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.724712_78.1","section":"bucket","name":"gpxthc-15","timestamp":"2020-06-04T08:29:31.724712205Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CRGlFFvUIxHRKMSeAVvS4kF","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.428231_79.1","section":"bucket.instance","name":"gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25","timestamp":"2020-06-04T08:30:16.428231743Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.433809_94.1","section":"bucket.instance","name":"gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25","timestamp":"2020-06-04T08:30:16.433809103Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.435876_95.1","section":"bucket","name":"gpxthc-16","timestamp":"2020-06-04T08:30:16.435876487Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4nNLQIaMSuVxPP12bYBinmN","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.440082_96.1","section":"bucket","name":"gpxthc-16","timestamp":"2020-06-04T08:30:16.440082082Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4nNLQIaMSuVxPP12bYBinmN","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.465814_97.1","section":"bucket.instance","name":"gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26","timestamp":"2020-06-04T08:30:16.465814342Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.470277_98.1","section":"bucket.instance","name":"gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26","timestamp":"2020-06-04T08:30:16.470277700Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.472163_99.1","section":"bucket","name":"gpxthc-17","timestamp":"2020-06-04T08:30:16.472163913Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jIqc-jlMoWLqci8_G-SuMb2","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.476487_100.1","section":"bucket","name":"gpxthc-17","timestamp":"2020-06-04T08:30:16.476487448Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jIqc-jlMoWLqci8_G-SuMb2","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.517171_101.1","section":"bucket.instance","name":"gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27","timestamp":"2020-06-04T08:30:16.517171806Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.522567_102.1","section":"bucket.instance","name":"gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27","timestamp":"2020-06-04T08:30:16.522567636Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.524592_103.1","section":"bucket","name":"gpxthc-18","timestamp":"2020-06-04T08:30:16.524592149Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T8PKYfLd2MscBs2JJLeNmbu","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.529893_104.1","section":"bucket","name":"gpxthc-18","timestamp":"2020-06-04T08:30:16.529893092Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T8PKYfLd2MscBs2JJLeNmbu","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.564959_105.1","section":"bucket.instance","name":"gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25","timestamp":"2020-06-04T08:30:16.564959314Z","data":{"read_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":1},"write_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":2},"status":{"status":"write"}}},{"id":"1_1591259416.569413_106.1","section":"bucket.instance","name":"gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25","timestamp":"2020-06-04T08:30:16.569413363Z","data":{"read_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":1},"write_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259416.573339_107.1","section":"bucket.instance","name":"gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26","timestamp":"2020-06-04T08:30:16.573339571Z","data":{"read_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":1},"write_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":2},"status":{"status":"write"}}},{"id":"1_1591259416.578468_108.1","section":"bucket.instance","name":"gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26","timestamp":"2020-06-04T08:30:16.578468486Z","data":{"read_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":1},"write_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259416.593653_109.1","section":"bucket.instance","name":"gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27","timestamp":"2020-06-04T08:30:16.593653914Z","data":{"read_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":1},"write_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":2},"status":{"status":"write"}}},{"id":"1_1591259416.599911_110.1","section":"bucket.instance","name":"gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27","timestamp":"2020-06-04T08:30:16.599911549Z","data":{"read_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":1},"write_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259628.323783_111.1","section":"bucket.instance","name":"gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28","timestamp":"2020-06-04T08:33:48.323783159Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":1},"status":{"status":"write"}}},{"id":"1_1591259628.328728_148.1","section":"bucket.instance","name":"gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28","timestamp":"2020-06-04T08:33:48.328728853Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259628.330710_149.1","section":"bucket","name":"gpxthc-19","timestamp":"2020-06-04T08:33:48.330710900Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_93QBHlBvEVUFT9b_wxOIMnq","ver":1},"status":{"status":"write"}}},{"id":"1_1591259628.334787_150.1","section":"bucket","name":"gpxthc-19","timestamp":"2020-06-04T08:33:48.334787040Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_93QBHlBvEVUFT9b_wxOIMnq","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259628.341176_152.1","section":"bucket.instance","name":"gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28","timestamp":"2020-06-04T08:33:48.341176479Z","data":{"read_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":1},"write_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":2},"status":{"status":"write"}}},{"id":"1_1591259628.347410_153.1","section":"bucket.instance","name":"gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28","timestamp":"2020-06-04T08:33:48.347410493Z","data":{"read_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":1},"write_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259767.841895_739.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:36:07.841895512Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":1},"status":{"status":"write"}}},{"id":"1_1591259767.846376_745.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:36:07.846376164Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259767.848773_746.1","section":"bucket","name":"gpxthc-20","timestamp":"2020-06-04T08:36:07.848773543Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ErTLIwy11AZgBe_n6j9EOvj","ver":1},"status":{"status":"write"}}},{"id":"1_1591259767.852808_747.1","section":"bucket","name":"gpxthc-20","timestamp":"2020-06-04T08:36:07.852808654Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ErTLIwy11AZgBe_n6j9EOvj","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259774.352869_748.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:36:14.352869429Z","data":{"read_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":1},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":2},"status":{"status":"write"}}},{"id":"1_1591259774.363917_755.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:36:14.363917038Z","data":{"read_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":1},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259833.992686_770.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:37:13.992686192Z","data":{"read_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":2},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":3},"status":{"status":"write"}}},{"id":"1_1591259834.004633_772.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:37:14.004633298Z","data":{"read_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":2},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":3},"status":{"status":"complete"}}},{"id":"1_1591259877.840447_773.1","section":"bucket.instance","name":"gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30","timestamp":"2020-06-04T08:37:57.840447936Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.844931_787.1","section":"bucket.instance","name":"gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30","timestamp":"2020-06-04T08:37:57.844931870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.847150_788.1","section":"bucket","name":"gpxthc-21","timestamp":"2020-06-04T08:37:57.847150642Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N959Mvlm6EfOalZxV2Ys2gP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.851202_789.1","section":"bucket","name":"gpxthc-21","timestamp":"2020-06-04T08:37:57.851202385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N959Mvlm6EfOalZxV2Ys2gP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.876704_790.1","section":"bucket.instance","name":"gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31","timestamp":"2020-06-04T08:37:57.876704072Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.881398_791.1","section":"bucket.instance","name":"gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31","timestamp":"2020-06-04T08:37:57.881398442Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.883373_792.1","section":"bucket","name":"gpxthc-22","timestamp":"2020-06-04T08:37:57.883373243Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dEQaV8gUPyD0QSgk_KOV4um","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.887399_793.1","section":"bucket","name":"gpxthc-22","timestamp":"2020-06-04T08:37:57.887399447Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dEQaV8gUPyD0QSgk_KOV4um","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.931319_794.1","section":"bucket.instance","name":"gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32","timestamp":"2020-06-04T08:37:57.931319262Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.936911_795.1","section":"bucket.instance","name":"gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32","timestamp":"2020-06-04T08:37:57.936911048Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.939469_796.1","section":"bucket","name":"gpxthc-23","timestamp":"2020-06-04T08:37:57.939469678Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c3fe5iwyxyrYzoUjllCpsvU","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.943713_797.1","section":"bucket","name":"gpxthc-23","timestamp":"2020-06-04T08:37:57.943713180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c3fe5iwyxyrYzoUjllCpsvU","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.969944_798.1","section":"bucket.instance","name":"gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30","timestamp":"2020-06-04T08:37:57.969944576Z","data":{"read_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":1},"write_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":2},"status":{"status":"write"}}},{"id":"1_1591259877.980272_799.1","section":"bucket.instance","name":"gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30","timestamp":"2020-06-04T08:37:57.980272398Z","data":{"read_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":1},"write_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259877.984533_800.1","section":"bucket.instance","name":"gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31","timestamp":"2020-06-04T08:37:57.984533518Z","data":{"read_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":1},"write_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":2},"status":{"status":"write"}}},{"id":"1_1591259877.989050_801.1","section":"bucket.instance","name":"gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31","timestamp":"2020-06-04T08:37:57.989050456Z","data":{"read_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":1},"write_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259878.000419_802.1","section":"bucket.instance","name":"gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32","timestamp":"2020-06-04T08:37:58.000419669Z","data":{"read_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":1},"write_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":2},"status":{"status":"write"}}},{"id":"1_1591259878.006196_803.1","section":"bucket.instance","name":"gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32","timestamp":"2020-06-04T08:37:58.006196070Z","data":{"read_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":1},"write_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259904.804937_804.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.804937045Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.810097_821.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.810097096Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.812274_822.1","section":"bucket","name":"gpxthc-24","timestamp":"2020-06-04T08:38:24.812274099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RQ8eJ9g70-tjyJRj0QIilAr","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.816235_823.1","section":"bucket","name":"gpxthc-24","timestamp":"2020-06-04T08:38:24.816235964Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RQ8eJ9g70-tjyJRj0QIilAr","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.846016_824.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:24.846016920Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.850663_825.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:24.850663462Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.852772_826.1","section":"bucket","name":"gpxthc-25","timestamp":"2020-06-04T08:38:24.852772198Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nSe4ph9lkroFUUb_S85OnlC","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.857137_827.1","section":"bucket","name":"gpxthc-25","timestamp":"2020-06-04T08:38:24.857137822Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nSe4ph9lkroFUUb_S85OnlC","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.906938_828.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:24.906938522Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.912058_829.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:24.912058812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.914274_830.1","section":"bucket","name":"gpxthc-26","timestamp":"2020-06-04T08:38:24.914274818Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_edlwuY9ktEE0aOm2248enxI","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.918216_831.1","section":"bucket","name":"gpxthc-26","timestamp":"2020-06-04T08:38:24.918216849Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_edlwuY9ktEE0aOm2248enxI","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.939673_832.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.939673512Z","data":{"read_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":1},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":2},"status":{"status":"write"}}},{"id":"1_1591259904.945900_833.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.945900417Z","data":{"read_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":1},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259904.952407_834.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.952407248Z","data":{"read_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":2},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":3},"status":{"status":"write"}}},{"id":"1_1591259904.959605_835.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.959605042Z","data":{"read_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":2},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":3},"status":{"status":"complete"}}},{"id":"1_1591259904.969679_836.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:24.969679222Z","data":{"read_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":1},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":2},"status":{"status":"write"}}},{"id":"1_1591259904.981104_837.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:24.981104198Z","data":{"read_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":1},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259905.015511_838.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:25.015511493Z","data":{"read_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":2},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":3},"status":{"status":"write"}}},{"id":"1_1591259905.023462_839.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:25.023462062Z","data":{"read_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":2},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":3},"status":{"status":"complete"}}},{"id":"1_1591259905.044506_840.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:25.044506828Z","data":{"read_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":1},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":2},"status":{"status":"write"}}},{"id":"1_1591259905.049929_841.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:25.049929340Z","data":{"read_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":1},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259905.058537_842.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:25.058537498Z","data":{"read_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":2},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":3},"status":{"status":"write"}}},{"id":"1_1591259905.064571_843.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:25.064571488Z","data":{"read_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":2},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":3},"status":{"status":"complete"}}},{"id":"1_1591259925.725606_844.1","section":"bucket.instance","name":"gpxthc-27:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.36","timestamp":"2020-06-04T08:38:45.725606301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.729923_861.1","section":"bucket.instance","name":"gpxthc-27:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.36","timestamp":"2020-06-04T08:38:45.729923565Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.732041_862.1","section":"bucket","name":"gpxthc-27","timestamp":"2020-06-04T08:38:45.732041328Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pBr-gACv7tk1mBrJwyLeMiI","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.736204_863.1","section":"bucket","name":"gpxthc-27","timestamp":"2020-06-04T08:38:45.736204921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pBr-gACv7tk1mBrJwyLeMiI","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.764022_864.1","section":"bucket.instance","name":"gpxthc-28:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.37","timestamp":"2020-06-04T08:38:45.764022725Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.768419_865.1","section":"bucket.instance","name":"gpxthc-28:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.37","timestamp":"2020-06-04T08:38:45.768419481Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.770593_866.1","section":"bucket","name":"gpxthc-28","timestamp":"2020-06-04T08:38:45.770593129Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YbhL-HGVrfTpz3aCSgkZ-1i","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.775197_867.1","section":"bucket","name":"gpxthc-28","timestamp":"2020-06-04T08:38:45.775197377Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YbhL-HGVrfTpz3aCSgkZ-1i","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.820803_868.1","section":"bucket.instance","name":"gpxthc-29:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.38","timestamp":"2020-06-04T08:38:45.820803858Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.825390_869.1","section":"bucket.instance","name":"gpxthc-29:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.38","timestamp":"2020-06-04T08:38:45.825390920Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.827444_870.1","section":"bucket","name":"gpxthc-29","timestamp":"2020-06-04T08:38:45.827444882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HJvkKg7YYuNuo_RQ2C1dEVG","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.831901_871.1","section":"bucket","name":"gpxthc-29","timestamp":"2020-06-04T08:38:45.831901352Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HJvkKg7YYuNuo_RQ2C1dEVG","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.856470_872.1","section":"bucket.instance","name":"gpxthc-27:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.36","timestamp":"2020-06-04T08:38:45.856470724Z","data":{"read_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":1},"write_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":2},"status":{"status":"write"}}},{"id":"1_1591259925.861409_873.1","section":"bucket.instance","name":"gpxthc-27:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.36","timestamp":"2020-06-04T08:38:45.861409642Z","data":{"read_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":1},"write_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259925.913880_874.1","section":"bucket.instance","name":"gpxthc-28:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.37","timestamp":"2020-06-04T08:38:45.913880548Z","data":{"read_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":1},"write_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":2},"status":{"status":"write"}}},{"id":"1_1591259925.924621_875.1","section":"bucket.instance","name":"gpxthc-28:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.37","timestamp":"2020-06-04T08:38:45.924621260Z","data":{"read_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":1},"write_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259925.983236_876.1","section":"bucket.instance","name":"gpxthc-29:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.38","timestamp":"2020-06-04T08:38:45.983236325Z","data":{"read_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":1},"write_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":2},"status":{"status":"write"}}},{"id":"1_1591259925.996695_877.1","section":"bucket.instance","name":"gpxthc-29:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.38","timestamp":"2020-06-04T08:38:45.996695629Z","data":{"read_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":1},"write_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259926.077449_878.1","section":"bucket.instance","name":"gpxthc-30:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.39","timestamp":"2020-06-04T08:38:46.077449382Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.083000_879.1","section":"bucket.instance","name":"gpxthc-30:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.39","timestamp":"2020-06-04T08:38:46.083000564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.085045_880.1","section":"bucket","name":"gpxthc-30","timestamp":"2020-06-04T08:38:46.085045274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jthV6t1d3f5V-3IMp3Qs5U2","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.088922_881.1","section":"bucket","name":"gpxthc-30","timestamp":"2020-06-04T08:38:46.088922963Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jthV6t1d3f5V-3IMp3Qs5U2","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.117432_883.1","section":"bucket.instance","name":"gpxthc-31:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.40","timestamp":"2020-06-04T08:38:46.117432535Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.121915_884.1","section":"bucket.instance","name":"gpxthc-31:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.40","timestamp":"2020-06-04T08:38:46.121915138Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.124124_885.1","section":"bucket","name":"gpxthc-31","timestamp":"2020-06-04T08:38:46.124124794Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6ku9WQrR5WvQ3xnFGpcCP0M","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.129078_886.1","section":"bucket","name":"gpxthc-31","timestamp":"2020-06-04T08:38:46.129078803Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6ku9WQrR5WvQ3xnFGpcCP0M","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.175739_887.1","section":"bucket.instance","name":"gpxthc-32:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.41","timestamp":"2020-06-04T08:38:46.175739838Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.182982_888.1","section":"bucket.instance","name":"gpxthc-32:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.41","timestamp":"2020-06-04T08:38:46.182982979Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.186641_889.1","section":"bucket","name":"gpxthc-32","timestamp":"2020-06-04T08:38:46.186641231Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RUtCJbRmBTaCDOrE5uNNCNT","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.193307_890.1","section":"bucket","name":"gpxthc-32","timestamp":"2020-06-04T08:38:46.193307183Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RUtCJbRmBTaCDOrE5uNNCNT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.371575_892.1","section":"bucket.instance","name":"gpxthc-30:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.39","timestamp":"2020-06-04T08:38:46.371575852Z","data":{"read_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":1},"write_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":2},"status":{"status":"write"}}},{"id":"1_1591259926.377365_893.1","section":"bucket.instance","name":"gpxthc-30:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.39","timestamp":"2020-06-04T08:38:46.377365569Z","data":{"read_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":1},"write_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259926.431027_896.1","section":"bucket.instance","name":"gpxthc-31:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.40","timestamp":"2020-06-04T08:38:46.431027842Z","data":{"read_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":1},"write_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":2},"status":{"status":"write"}}},{"id":"1_1591259926.442905_897.1","section":"bucket.instance","name":"gpxthc-31:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.40","timestamp":"2020-06-04T08:38:46.442905741Z","data":{"read_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":1},"write_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259926.503230_898.1","section":"bucket.instance","name":"gpxthc-32:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.41","timestamp":"2020-06-04T08:38:46.503230913Z","data":{"read_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":1},"write_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":2},"status":{"status":"write"}}},{"id":"1_1591259926.513386_899.1","section":"bucket.instance","name":"gpxthc-32:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.41","timestamp":"2020-06-04T08:38:46.513386815Z","data":{"read_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":1},"write_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259926.597611_900.1","section":"bucket.instance","name":"gpxthc-33:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.42","timestamp":"2020-06-04T08:38:46.597611762Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.604278_901.1","section":"bucket.instance","name":"gpxthc-33:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.42","timestamp":"2020-06-04T08:38:46.604278594Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.611816_902.1","section":"bucket","name":"gpxthc-33","timestamp":"2020-06-04T08:38:46.611816568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TCD6G-8d3BbB8YgWl8O3Nwk","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.616028_903.1","section":"bucket","name":"gpxthc-33","timestamp":"2020-06-04T08:38:46.616028597Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TCD6G-8d3BbB8YgWl8O3Nwk","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.637171_904.1","section":"bucket.instance","name":"gpxthc-34:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.43","timestamp":"2020-06-04T08:38:46.637171077Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.641863_905.1","section":"bucket.instance","name":"gpxthc-34:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.43","timestamp":"2020-06-04T08:38:46.641863656Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.644434_906.1","section":"bucket","name":"gpxthc-34","timestamp":"2020-06-04T08:38:46.644434253Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P-m8zGy2fOnlsKo8No6u1Cq","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.652520_907.1","section":"bucket","name":"gpxthc-34","timestamp":"2020-06-04T08:38:46.652520435Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P-m8zGy2fOnlsKo8No6u1Cq","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.695989_908.1","section":"bucket.instance","name":"gpxthc-35:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.44","timestamp":"2020-06-04T08:38:46.695989087Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.700896_909.1","section":"bucket.instance","name":"gpxthc-35:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.44","timestamp":"2020-06-04T08:38:46.700896824Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.703158_910.1","section":"bucket","name":"gpxthc-35","timestamp":"2020-06-04T08:38:46.703158174Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gEvrMnPJFZrfyPUGCekqix_","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.707672_911.1","section":"bucket","name":"gpxthc-35","timestamp":"2020-06-04T08:38:46.707672554Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gEvrMnPJFZrfyPUGCekqix_","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.732054_912.1","section":"bucket.instance","name":"gpxthc-33:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.42","timestamp":"2020-06-04T08:38:46.732054711Z","data":{"read_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":1},"write_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":2},"status":{"status":"write"}}},{"id":"1_1591259926.737279_913.1","section":"bucket.instance","name":"gpxthc-33:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.42","timestamp":"2020-06-04T08:38:46.737279135Z","data":{"read_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":1},"write_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259927.790025_914.1","section":"bucket.instance","name":"gpxthc-34:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.43","timestamp":"2020-06-04T08:38:47.790025773Z","data":{"read_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":1},"write_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":2},"status":{"status":"write"}}},{"id":"1_1591259927.802447_915.1","section":"bucket.instance","name":"gpxthc-34:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.43","timestamp":"2020-06-04T08:38:47.802447374Z","data":{"read_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":1},"write_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259928.861627_916.1","section":"bucket.instance","name":"gpxthc-35:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.44","timestamp":"2020-06-04T08:38:48.861627553Z","data":{"read_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":1},"write_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":2},"status":{"status":"write"}}},{"id":"1_1591259928.873592_917.1","section":"bucket.instance","name":"gpxthc-35:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.44","timestamp":"2020-06-04T08:38:48.873592348Z","data":{"read_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":1},"write_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259928.954520_918.1","section":"bucket.instance","name":"gpxthc-36:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.45","timestamp":"2020-06-04T08:38:48.954520125Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_11-lNmUdWJkycuWy12Rkais","ver":1},"status":{"status":"write"}}},{"id":"1_1591259928.958725_919.1","section":"bucket.instance","name":"gpxthc-36:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.45","timestamp":"2020-06-04T08:38:48.958725871Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_11-lNmUdWJkycuWy12Rkais","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259928.960607_920.1","section":"bucket","name":"gpxthc-36","timestamp":"2020-06-04T08:38:48.960607582Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2lIEwet-R7Pmma_XC0XfJfn","ver":1},"status":{"status":"write"}}},{"id":"1_1591259928.964925_921.1","section":"bucket","name":"gpxthc-36","timestamp":"2020-06-04T08:38:48.964925190Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2lIEwet-R7Pmma_XC0XfJfn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259929.002482_922.1","section":"bucket.instance","name":"gpxthc-37:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.46","timestamp":"2020-06-04T08:38:49.002482783Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V-CEvpx3s4VyGthwNmj1yAU","ver":1},"status":{"status":"write"}}},{"id":"1_1591259929.007285_923.1","section":"bucket.instance","name":"gpxthc-37:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.46","timestamp":"2020-06-04T08:38:49.007285334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V-CEvpx3s4VyGthwNmj1yAU","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259929.009398_924.1","section":"bucket","name":"gpxthc-37","timestamp":"2020-06-04T08:38:49.009398457Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ek_-Wa_Uyu-XnjfASqAxdXy","ver":1},"status":{"status":"write"}}},{"id":"1_1591259929.013412_925.1","section":"bucket","name":"gpxthc-37","timestamp":"2020-06-04T08:38:49.013412886Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ek_-Wa_Uyu-XnjfASqAxdXy","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259929.052573_926.1","section":"bucket.instance","name":"gpxthc-38:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.47","timestamp":"2020-06-04T08:38:49.052573944Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N6EHuJJFZ7_2dQ30ZsrYxhQ","ver":1},"status":{"status":"write"}}},{"id":"1_1591259929.056846_927.1","section":"bucket.instance","name":"gpxthc-38:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.47","timestamp":"2020-06-04T08:38:49.056846763Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N6EHuJJFZ7_2dQ30ZsrYxhQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259929.059064_928.1","section":"bucket","name":"gpxthc-38","timestamp":"2020-06-04T08:38:49.059064075Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-zv4Cksv-zkyzdIqRjsaDb2","ver":1},"status":{"status":"write"}}},{"id":"1_1591259929.063266_929.1","section":"bucket","name":"gpxthc-38","timestamp":"2020-06-04T08:38:49.063266714Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-zv4Cksv-zkyzdIqRjsaDb2","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.609274_930.1","section":"bucket.instance","name":"gpxthc-39:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.48","timestamp":"2020-06-04T08:38:50.609274737Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VhICsf9zqk7f9jpddSNIqkP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.613351_932.1","section":"bucket.instance","name":"gpxthc-39:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.48","timestamp":"2020-06-04T08:38:50.613351600Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VhICsf9zqk7f9jpddSNIqkP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.615477_933.1","section":"bucket","name":"gpxthc-39","timestamp":"2020-06-04T08:38:50.615477264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v902jpCGpnCkP2gKKzOXdqt","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.619346_934.1","section":"bucket","name":"gpxthc-39","timestamp":"2020-06-04T08:38:50.619346741Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v902jpCGpnCkP2gKKzOXdqt","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.653610_935.1","section":"bucket.instance","name":"gpxthc-40:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.49","timestamp":"2020-06-04T08:38:50.653610015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pbAYVMP2sAf_BuaE5BYBTkr","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.658862_936.1","section":"bucket.instance","name":"gpxthc-40:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.49","timestamp":"2020-06-04T08:38:50.658862411Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pbAYVMP2sAf_BuaE5BYBTkr","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.661191_937.1","section":"bucket","name":"gpxthc-40","timestamp":"2020-06-04T08:38:50.661191135Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4cw4XeidBA2TPzMfHBQhnsn","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.665356_938.1","section":"bucket","name":"gpxthc-40","timestamp":"2020-06-04T08:38:50.665356122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4cw4XeidBA2TPzMfHBQhnsn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.711577_939.1","section":"bucket.instance","name":"gpxthc-41:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.50","timestamp":"2020-06-04T08:38:50.711577903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zFKqF0DR8qyD3-Tr3FbcjW1","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.719322_940.1","section":"bucket.instance","name":"gpxthc-41:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.50","timestamp":"2020-06-04T08:38:50.719322092Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zFKqF0DR8qyD3-Tr3FbcjW1","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.721242_941.1","section":"bucket","name":"gpxthc-41","timestamp":"2020-06-04T08:38:50.721242218Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n_mX6j5IFhQpmY51w3jLeA3","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.725192_942.1","section":"bucket","name":"gpxthc-41","timestamp":"2020-06-04T08:38:50.725192405Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n_mX6j5IFhQpmY51w3jLeA3","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259341.737046_34.1","section":"bucket.instance","name":"gpxthc-1:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.1","timestamp":"2020-06-04T08:29:01.737046909Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-VXJYi2_CLvrsqw6qc6DCr5","ver":1},"status":{"status":"write"}}},{"id":"1_1591259341.740440_35.1","section":"bucket.instance","name":"gpxthc-1:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.1","timestamp":"2020-06-04T08:29:01.740440081Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-VXJYi2_CLvrsqw6qc6DCr5","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259341.742105_36.1","section":"bucket","name":"gpxthc-1","timestamp":"2020-06-04T08:29:01.742105318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6GkvMPsGVWRJghpjcfPP5td","ver":1},"status":{"status":"write"}}},{"id":"1_1591259341.745009_37.1","section":"bucket","name":"gpxthc-1","timestamp":"2020-06-04T08:29:01.745009929Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6GkvMPsGVWRJghpjcfPP5td","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259341.768025_38.1","section":"bucket.instance","name":"gpxthc-2:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.2","timestamp":"2020-06-04T08:29:01.768025171Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XvNT082cvaVbqbr-2ZESWeT","ver":1},"status":{"status":"write"}}},{"id":"1_1591259341.771975_39.1","section":"bucket.instance","name":"gpxthc-2:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.2","timestamp":"2020-06-04T08:29:01.771975782Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XvNT082cvaVbqbr-2ZESWeT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259341.773834_40.1","section":"bucket","name":"gpxthc-2","timestamp":"2020-06-04T08:29:01.773834414Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AQGxlKt5b4yEuqgCWbWM_je","ver":1},"status":{"status":"write"}}},{"id":"1_1591259341.777323_41.1","section":"bucket","name":"gpxthc-2","timestamp":"2020-06-04T08:29:01.777323422Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AQGxlKt5b4yEuqgCWbWM_je","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259345.200120_42.1","section":"bucket.instance","name":"gpxthc-3:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.3","timestamp":"2020-06-04T08:29:05.200120296Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r--8CKmcPrYbCzDhO-F06sP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259345.203880_43.1","section":"bucket.instance","name":"gpxthc-3:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.3","timestamp":"2020-06-04T08:29:05.203880111Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r--8CKmcPrYbCzDhO-F06sP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259345.205531_44.1","section":"bucket","name":"gpxthc-3","timestamp":"2020-06-04T08:29:05.205531158Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bgP8ilt4JXWshxEQEKDTOxL","ver":1},"status":{"status":"write"}}},{"id":"1_1591259345.209069_45.1","section":"bucket","name":"gpxthc-3","timestamp":"2020-06-04T08:29:05.209069471Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bgP8ilt4JXWshxEQEKDTOxL","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.383145_46.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.4","timestamp":"2020-06-04T08:29:06.383145515Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HbOqQVlA66qYh_U-6gTxraM","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.388144_47.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.4","timestamp":"2020-06-04T08:29:06.388144053Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HbOqQVlA66qYh_U-6gTxraM","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.390123_48.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:06.390123332Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ThXePFlF36ZdSx-rFItfhQ8","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.394528_49.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:06.394528126Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ThXePFlF36ZdSx-rFItfhQ8","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.415142_50.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.5","timestamp":"2020-06-04T08:29:06.415142498Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FgaMM_80CprME0J7uTE6atP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.419201_51.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.5","timestamp":"2020-06-04T08:29:06.419201114Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FgaMM_80CprME0J7uTE6atP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.421086_52.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:06.421086802Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Nj0SyzQoWhfLBcf7quWXETq","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.424778_53.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:06.424778499Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Nj0SyzQoWhfLBcf7quWXETq","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.463965_54.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.6","timestamp":"2020-06-04T08:29:06.463965385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-cllnOXQdxzDqATQ1dbWkmx","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.467779_55.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.6","timestamp":"2020-06-04T08:29:06.467779786Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-cllnOXQdxzDqATQ1dbWkmx","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.469502_56.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:06.469502201Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uaSCzOL2DVEOK-odgsLCwFs","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.473034_57.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:06.473034822Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uaSCzOL2DVEOK-odgsLCwFs","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.614173_58.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.7","timestamp":"2020-06-04T08:29:07.614173009Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QW7H8a3CUThQ_bRH4yAqRKv","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.618181_59.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.7","timestamp":"2020-06-04T08:29:07.618181728Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QW7H8a3CUThQ_bRH4yAqRKv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.619881_60.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:07.619881218Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5sgvOjv3iB27roqBQ7cUZbh","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.623348_61.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:07.623348297Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5sgvOjv3iB27roqBQ7cUZbh","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259347.647460_62.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.8","timestamp":"2020-06-04T08:29:07.647460278Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eGGXzzZG1A9GEn7uWckVfr-","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.651908_63.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.8","timestamp":"2020-06-04T08:29:07.651908533Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eGGXzzZG1A9GEn7uWckVfr-","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.654187_64.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:07.654187090Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UJzl8Whz1V5QSBu3ohXNupT","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.657537_65.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:07.657537293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UJzl8Whz1V5QSBu3ohXNupT","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259347.686764_66.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.9","timestamp":"2020-06-04T08:29:07.686764572Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vZquMGd6hFx42B6ADczwHim","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.691403_67.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.9","timestamp":"2020-06-04T08:29:07.691403042Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vZquMGd6hFx42B6ADczwHim","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.693513_68.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:07.693513613Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UOipH4LH_UUuQBoXOTlPqdc","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.698070_69.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:07.698070773Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UOipH4LH_UUuQBoXOTlPqdc","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259347.733629_70.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.10","timestamp":"2020-06-04T08:29:07.733629268Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4NNS9aPc6N_RBBTATY5XJN5","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.737752_71.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.10","timestamp":"2020-06-04T08:29:07.737752160Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4NNS9aPc6N_RBBTATY5XJN5","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.739454_72.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:07.739454023Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9rrBHViPIaHB1oKA4-fy2a7","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.742970_73.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:07.742970942Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9rrBHViPIaHB1oKA4-fy2a7","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259347.792004_74.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.11","timestamp":"2020-06-04T08:29:07.792004579Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R8wjaedoiarBcdJxSdsNG9i","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.797167_75.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.11","timestamp":"2020-06-04T08:29:07.797167708Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R8wjaedoiarBcdJxSdsNG9i","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.798799_76.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:07.798799372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFqu-tIqXwMndR9GZsMEdUg","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.801827_77.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:07.801827701Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFqu-tIqXwMndR9GZsMEdUg","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259347.895642_78.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.12","timestamp":"2020-06-04T08:29:07.895642957Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_813CHCsdbVJhG-8Friwea0Q","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.900475_79.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.12","timestamp":"2020-06-04T08:29:07.900475349Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_813CHCsdbVJhG-8Friwea0Q","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.903751_80.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:07.903751639Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4jTRpPwqpn_U-wRoBFCv-5t","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.907394_81.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:07.907394318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4jTRpPwqpn_U-wRoBFCv-5t","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259348.003913_82.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.13","timestamp":"2020-06-04T08:29:08.003913702Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vjl-KiOIIS-mHj4Knkvyvbn","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.008410_83.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.13","timestamp":"2020-06-04T08:29:08.008410230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vjl-KiOIIS-mHj4Knkvyvbn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259348.010405_84.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:08.010405065Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YXdPZK993bwZgNRYKqmPKbg","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.013902_85.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:08.013902468Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YXdPZK993bwZgNRYKqmPKbg","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259348.061824_86.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.14","timestamp":"2020-06-04T08:29:08.061824583Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GmPkZfz9onYDYux9NvroVpb","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.066453_87.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.14","timestamp":"2020-06-04T08:29:08.066453149Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GmPkZfz9onYDYux9NvroVpb","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259348.068278_88.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:08.068278859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n5UFL0MW6wtQDqqBX5p9xOv","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.071562_89.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:08.071562898Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n5UFL0MW6wtQDqqBX5p9xOv","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259348.164678_90.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.15","timestamp":"2020-06-04T08:29:08.164678779Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wnNQvYf5A6zQTv_L1L16Vj_","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.169209_91.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.15","timestamp":"2020-06-04T08:29:08.169209509Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wnNQvYf5A6zQTv_L1L16Vj_","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259348.171482_92.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:08.171482220Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HuczynHZJw9zrk53biUsBon","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.175359_93.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:08.175359125Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HuczynHZJw9zrk53biUsBon","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259349.444497_94.1","section":"bucket.instance","name":"gpxthc-7:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.16","timestamp":"2020-06-04T08:29:09.444497949Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0hH3KdNQ7Ror_-PbjMomrOK","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.449423_95.1","section":"bucket.instance","name":"gpxthc-7:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.16","timestamp":"2020-06-04T08:29:09.449423359Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0hH3KdNQ7Ror_-PbjMomrOK","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259349.451676_96.1","section":"bucket","name":"gpxthc-7","timestamp":"2020-06-04T08:29:09.451676822Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.455775_97.1","section":"bucket","name":"gpxthc-7","timestamp":"2020-06-04T08:29:09.455775359Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259349.478408_98.1","section":"bucket.instance","name":"gpxthc-8:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.17","timestamp":"2020-06-04T08:29:09.478408677Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3tdK5SOwUxP7sixtd5Iiusu","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.483441_99.1","section":"bucket.instance","name":"gpxthc-8:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.17","timestamp":"2020-06-04T08:29:09.483441043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3tdK5SOwUxP7sixtd5Iiusu","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259349.485210_100.1","section":"bucket","name":"gpxthc-8","timestamp":"2020-06-04T08:29:09.485210362Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.489008_101.1","section":"bucket","name":"gpxthc-8","timestamp":"2020-06-04T08:29:09.489008069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259349.542500_102.1","section":"bucket.instance","name":"gpxthc-9:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.18","timestamp":"2020-06-04T08:29:09.542500556Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZM4c0G2TMLKguEZQHJhJ_U1","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.547391_103.1","section":"bucket.instance","name":"gpxthc-9:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.18","timestamp":"2020-06-04T08:29:09.547391664Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZM4c0G2TMLKguEZQHJhJ_U1","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259349.549662_104.1","section":"bucket","name":"gpxthc-9","timestamp":"2020-06-04T08:29:09.549662472Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.557874_105.1","section":"bucket","name":"gpxthc-9","timestamp":"2020-06-04T08:29:09.557874857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259350.809493_106.1","section":"bucket","name":"gpxthc-7","timestamp":"2020-06-04T08:29:10.809493171Z","data":{"read_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":1},"write_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":2},"status":{"status":"remove"}}},{"id":"1_1591259350.815979_107.1","section":"bucket","name":"gpxthc-7","timestamp":"2020-06-04T08:29:10.815979805Z","data":{"read_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":1},"write_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259350.882584_108.1","section":"bucket","name":"gpxthc-8","timestamp":"2020-06-04T08:29:10.882584350Z","data":{"read_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":1},"write_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":2},"status":{"status":"remove"}}},{"id":"1_1591259350.890059_109.1","section":"bucket","name":"gpxthc-8","timestamp":"2020-06-04T08:29:10.890059979Z","data":{"read_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":1},"write_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259350.978944_110.1","section":"bucket","name":"gpxthc-9","timestamp":"2020-06-04T08:29:10.978944722Z","data":{"read_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":1},"write_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":2},"status":{"status":"remove"}}},{"id":"1_1591259350.985454_111.1","section":"bucket","name":"gpxthc-9","timestamp":"2020-06-04T08:29:10.985454315Z","data":{"read_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":1},"write_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":2},"status":{"status":"complete"}}}] 2020-06-04T08:41:13.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '838ed6b2-fa2f-4917-a5c1-6c82bb405cda', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:13.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '838ed6b2-fa2f-4917-a5c1-6c82bb405cda', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:13.158 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 838ed6b2-fa2f-4917-a5c1-6c82bb405cda --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:13.197 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:13.197 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:13.460 INFO:teuthology.orchestra.run.smithi196.stdout:[{"id":"1_1591259219.822557_39.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-06-04T08:26:59.822557133Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UKgIH850zXzbqniMBbCDHLr","ver":1},"status":{"status":"write"}}},{"id":"1_1591259219.830770_41.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-06-04T08:26:59.830770292Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UKgIH850zXzbqniMBbCDHLr","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.183102_42.1","section":"bucket.instance","name":"gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19","timestamp":"2020-06-04T08:29:12.183102477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VfH48PXmsZggFiAsqBANxvt","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.187271_50.1","section":"bucket.instance","name":"gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19","timestamp":"2020-06-04T08:29:12.187271885Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VfH48PXmsZggFiAsqBANxvt","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.189031_51.1","section":"bucket","name":"gpxthc-10","timestamp":"2020-06-04T08:29:12.189031257Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_h8WUqJo-QErjBMZIQdgNCtn","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.192886_52.1","section":"bucket","name":"gpxthc-10","timestamp":"2020-06-04T08:29:12.192886901Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_h8WUqJo-QErjBMZIQdgNCtn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.214300_53.1","section":"bucket.instance","name":"gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20","timestamp":"2020-06-04T08:29:12.214300618Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OITM9jIqMd3jh6APsdWU31A","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.219348_54.1","section":"bucket.instance","name":"gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20","timestamp":"2020-06-04T08:29:12.219348485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OITM9jIqMd3jh6APsdWU31A","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.221278_55.1","section":"bucket","name":"gpxthc-11","timestamp":"2020-06-04T08:29:12.221278412Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-nUpFbAACCD5rYXC7DcxUQR","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.225194_56.1","section":"bucket","name":"gpxthc-11","timestamp":"2020-06-04T08:29:12.225194355Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-nUpFbAACCD5rYXC7DcxUQR","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.268612_57.1","section":"bucket.instance","name":"gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21","timestamp":"2020-06-04T08:29:12.268612619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zd8ii08n-ZihNb68aNArIKM","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.273166_58.1","section":"bucket.instance","name":"gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21","timestamp":"2020-06-04T08:29:12.273166922Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zd8ii08n-ZihNb68aNArIKM","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.275116_59.1","section":"bucket","name":"gpxthc-12","timestamp":"2020-06-04T08:29:12.275116572Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AleC4XiO8GV6Mta70FMehuN","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.279189_60.1","section":"bucket","name":"gpxthc-12","timestamp":"2020-06-04T08:29:12.279189055Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AleC4XiO8GV6Mta70FMehuN","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.619667_61.1","section":"bucket.instance","name":"gpxthc-13:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.22","timestamp":"2020-06-04T08:29:31.619667228Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G4zU5SP7vzOavY4UDRy0kVT","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.624339_68.1","section":"bucket.instance","name":"gpxthc-13:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.22","timestamp":"2020-06-04T08:29:31.624339358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G4zU5SP7vzOavY4UDRy0kVT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.626643_69.1","section":"bucket","name":"gpxthc-13","timestamp":"2020-06-04T08:29:31.626643743Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VUOAgr-b5NnlCfY5DmWhaqq","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.630935_70.1","section":"bucket","name":"gpxthc-13","timestamp":"2020-06-04T08:29:31.630935737Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VUOAgr-b5NnlCfY5DmWhaqq","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.653316_71.1","section":"bucket.instance","name":"gpxthc-14:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.23","timestamp":"2020-06-04T08:29:31.653316611Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YhqpqT7kUaddbo1-1KfMhvv","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.658400_72.1","section":"bucket.instance","name":"gpxthc-14:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.23","timestamp":"2020-06-04T08:29:31.658400301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YhqpqT7kUaddbo1-1KfMhvv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.660536_73.1","section":"bucket","name":"gpxthc-14","timestamp":"2020-06-04T08:29:31.660536384Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Br3NXT9kNBdoMbsZqQdcMrv","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.664400_74.1","section":"bucket","name":"gpxthc-14","timestamp":"2020-06-04T08:29:31.664400106Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Br3NXT9kNBdoMbsZqQdcMrv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.712373_75.1","section":"bucket.instance","name":"gpxthc-15:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.24","timestamp":"2020-06-04T08:29:31.712373958Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_frk3BKkQ_Qa4KdotA_oTnm0","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.716427_76.1","section":"bucket.instance","name":"gpxthc-15:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.24","timestamp":"2020-06-04T08:29:31.716427746Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_frk3BKkQ_Qa4KdotA_oTnm0","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.719782_77.1","section":"bucket","name":"gpxthc-15","timestamp":"2020-06-04T08:29:31.719782440Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CRGlFFvUIxHRKMSeAVvS4kF","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.724712_78.1","section":"bucket","name":"gpxthc-15","timestamp":"2020-06-04T08:29:31.724712205Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CRGlFFvUIxHRKMSeAVvS4kF","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.428231_79.1","section":"bucket.instance","name":"gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25","timestamp":"2020-06-04T08:30:16.428231743Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.433809_94.1","section":"bucket.instance","name":"gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25","timestamp":"2020-06-04T08:30:16.433809103Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.435876_95.1","section":"bucket","name":"gpxthc-16","timestamp":"2020-06-04T08:30:16.435876487Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4nNLQIaMSuVxPP12bYBinmN","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.440082_96.1","section":"bucket","name":"gpxthc-16","timestamp":"2020-06-04T08:30:16.440082082Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4nNLQIaMSuVxPP12bYBinmN","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.465814_97.1","section":"bucket.instance","name":"gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26","timestamp":"2020-06-04T08:30:16.465814342Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.470277_98.1","section":"bucket.instance","name":"gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26","timestamp":"2020-06-04T08:30:16.470277700Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.472163_99.1","section":"bucket","name":"gpxthc-17","timestamp":"2020-06-04T08:30:16.472163913Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jIqc-jlMoWLqci8_G-SuMb2","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.476487_100.1","section":"bucket","name":"gpxthc-17","timestamp":"2020-06-04T08:30:16.476487448Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jIqc-jlMoWLqci8_G-SuMb2","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.517171_101.1","section":"bucket.instance","name":"gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27","timestamp":"2020-06-04T08:30:16.517171806Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.522567_102.1","section":"bucket.instance","name":"gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27","timestamp":"2020-06-04T08:30:16.522567636Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.524592_103.1","section":"bucket","name":"gpxthc-18","timestamp":"2020-06-04T08:30:16.524592149Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T8PKYfLd2MscBs2JJLeNmbu","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.529893_104.1","section":"bucket","name":"gpxthc-18","timestamp":"2020-06-04T08:30:16.529893092Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T8PKYfLd2MscBs2JJLeNmbu","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.564959_105.1","section":"bucket.instance","name":"gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25","timestamp":"2020-06-04T08:30:16.564959314Z","data":{"read_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":1},"write_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":2},"status":{"status":"write"}}},{"id":"1_1591259416.569413_106.1","section":"bucket.instance","name":"gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25","timestamp":"2020-06-04T08:30:16.569413363Z","data":{"read_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":1},"write_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259416.573339_107.1","section":"bucket.instance","name":"gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26","timestamp":"2020-06-04T08:30:16.573339571Z","data":{"read_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":1},"write_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":2},"status":{"status":"write"}}},{"id":"1_1591259416.578468_108.1","section":"bucket.instance","name":"gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26","timestamp":"2020-06-04T08:30:16.578468486Z","data":{"read_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":1},"write_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259416.593653_109.1","section":"bucket.instance","name":"gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27","timestamp":"2020-06-04T08:30:16.593653914Z","data":{"read_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":1},"write_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":2},"status":{"status":"write"}}},{"id":"1_1591259416.599911_110.1","section":"bucket.instance","name":"gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27","timestamp":"2020-06-04T08:30:16.599911549Z","data":{"read_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":1},"write_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259628.323783_111.1","section":"bucket.instance","name":"gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28","timestamp":"2020-06-04T08:33:48.323783159Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":1},"status":{"status":"write"}}},{"id":"1_1591259628.328728_148.1","section":"bucket.instance","name":"gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28","timestamp":"2020-06-04T08:33:48.328728853Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259628.330710_149.1","section":"bucket","name":"gpxthc-19","timestamp":"2020-06-04T08:33:48.330710900Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_93QBHlBvEVUFT9b_wxOIMnq","ver":1},"status":{"status":"write"}}},{"id":"1_1591259628.334787_150.1","section":"bucket","name":"gpxthc-19","timestamp":"2020-06-04T08:33:48.334787040Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_93QBHlBvEVUFT9b_wxOIMnq","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259628.341176_152.1","section":"bucket.instance","name":"gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28","timestamp":"2020-06-04T08:33:48.341176479Z","data":{"read_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":1},"write_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":2},"status":{"status":"write"}}},{"id":"1_1591259628.347410_153.1","section":"bucket.instance","name":"gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28","timestamp":"2020-06-04T08:33:48.347410493Z","data":{"read_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":1},"write_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259767.841895_739.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:36:07.841895512Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":1},"status":{"status":"write"}}},{"id":"1_1591259767.846376_745.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:36:07.846376164Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259767.848773_746.1","section":"bucket","name":"gpxthc-20","timestamp":"2020-06-04T08:36:07.848773543Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ErTLIwy11AZgBe_n6j9EOvj","ver":1},"status":{"status":"write"}}},{"id":"1_1591259767.852808_747.1","section":"bucket","name":"gpxthc-20","timestamp":"2020-06-04T08:36:07.852808654Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ErTLIwy11AZgBe_n6j9EOvj","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259774.352869_748.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:36:14.352869429Z","data":{"read_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":1},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":2},"status":{"status":"write"}}},{"id":"1_1591259774.363917_755.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:36:14.363917038Z","data":{"read_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":1},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259833.992686_770.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:37:13.992686192Z","data":{"read_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":2},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":3},"status":{"status":"write"}}},{"id":"1_1591259834.004633_772.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:37:14.004633298Z","data":{"read_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":2},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":3},"status":{"status":"complete"}}},{"id":"1_1591259877.840447_773.1","section":"bucket.instance","name":"gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30","timestamp":"2020-06-04T08:37:57.840447936Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.844931_787.1","section":"bucket.instance","name":"gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30","timestamp":"2020-06-04T08:37:57.844931870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.847150_788.1","section":"bucket","name":"gpxthc-21","timestamp":"2020-06-04T08:37:57.847150642Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N959Mvlm6EfOalZxV2Ys2gP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.851202_789.1","section":"bucket","name":"gpxthc-21","timestamp":"2020-06-04T08:37:57.851202385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N959Mvlm6EfOalZxV2Ys2gP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.876704_790.1","section":"bucket.instance","name":"gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31","timestamp":"2020-06-04T08:37:57.876704072Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.881398_791.1","section":"bucket.instance","name":"gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31","timestamp":"2020-06-04T08:37:57.881398442Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.883373_792.1","section":"bucket","name":"gpxthc-22","timestamp":"2020-06-04T08:37:57.883373243Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dEQaV8gUPyD0QSgk_KOV4um","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.887399_793.1","section":"bucket","name":"gpxthc-22","timestamp":"2020-06-04T08:37:57.887399447Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dEQaV8gUPyD0QSgk_KOV4um","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.931319_794.1","section":"bucket.instance","name":"gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32","timestamp":"2020-06-04T08:37:57.931319262Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.936911_795.1","section":"bucket.instance","name":"gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32","timestamp":"2020-06-04T08:37:57.936911048Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.939469_796.1","section":"bucket","name":"gpxthc-23","timestamp":"2020-06-04T08:37:57.939469678Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c3fe5iwyxyrYzoUjllCpsvU","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.943713_797.1","section":"bucket","name":"gpxthc-23","timestamp":"2020-06-04T08:37:57.943713180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c3fe5iwyxyrYzoUjllCpsvU","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.969944_798.1","section":"bucket.instance","name":"gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30","timestamp":"2020-06-04T08:37:57.969944576Z","data":{"read_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":1},"write_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":2},"status":{"status":"write"}}},{"id":"1_1591259877.980272_799.1","section":"bucket.instance","name":"gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30","timestamp":"2020-06-04T08:37:57.980272398Z","data":{"read_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":1},"write_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259877.984533_800.1","section":"bucket.instance","name":"gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31","timestamp":"2020-06-04T08:37:57.984533518Z","data":{"read_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":1},"write_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":2},"status":{"status":"write"}}},{"id":"1_1591259877.989050_801.1","section":"bucket.instance","name":"gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31","timestamp":"2020-06-04T08:37:57.989050456Z","data":{"read_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":1},"write_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259878.000419_802.1","section":"bucket.instance","name":"gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32","timestamp":"2020-06-04T08:37:58.000419669Z","data":{"read_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":1},"write_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":2},"status":{"status":"write"}}},{"id":"1_1591259878.006196_803.1","section":"bucket.instance","name":"gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32","timestamp":"2020-06-04T08:37:58.006196070Z","data":{"read_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":1},"write_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259904.804937_804.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.804937045Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.810097_821.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.810097096Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.812274_822.1","section":"bucket","name":"gpxthc-24","timestamp":"2020-06-04T08:38:24.812274099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RQ8eJ9g70-tjyJRj0QIilAr","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.816235_823.1","section":"bucket","name":"gpxthc-24","timestamp":"2020-06-04T08:38:24.816235964Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RQ8eJ9g70-tjyJRj0QIilAr","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.846016_824.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:24.846016920Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.850663_825.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:24.850663462Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.852772_826.1","section":"bucket","name":"gpxthc-25","timestamp":"2020-06-04T08:38:24.852772198Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nSe4ph9lkroFUUb_S85OnlC","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.857137_827.1","section":"bucket","name":"gpxthc-25","timestamp":"2020-06-04T08:38:24.857137822Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nSe4ph9lkroFUUb_S85OnlC","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.906938_828.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:24.906938522Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.912058_829.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:24.912058812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.914274_830.1","section":"bucket","name":"gpxthc-26","timestamp":"2020-06-04T08:38:24.914274818Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_edlwuY9ktEE0aOm2248enxI","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.918216_831.1","section":"bucket","name":"gpxthc-26","timestamp":"2020-06-04T08:38:24.918216849Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_edlwuY9ktEE0aOm2248enxI","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.939673_832.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.939673512Z","data":{"read_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":1},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":2},"status":{"status":"write"}}},{"id":"1_1591259904.945900_833.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.945900417Z","data":{"read_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":1},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259904.952407_834.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.952407248Z","data":{"read_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":2},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":3},"status":{"status":"write"}}},{"id":"1_1591259904.959605_835.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.959605042Z","data":{"read_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":2},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":3},"status":{"status":"complete"}}},{"id":"1_1591259904.969679_836.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:24.969679222Z","data":{"read_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":1},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":2},"status":{"status":"write"}}},{"id":"1_1591259904.981104_837.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:24.981104198Z","data":{"read_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":1},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259905.015511_838.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:25.015511493Z","data":{"read_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":2},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":3},"status":{"status":"write"}}},{"id":"1_1591259905.023462_839.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:25.023462062Z","data":{"read_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":2},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":3},"status":{"status":"complete"}}},{"id":"1_1591259905.044506_840.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:25.044506828Z","data":{"read_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":1},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":2},"status":{"status":"write"}}},{"id":"1_1591259905.049929_841.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:25.049929340Z","data":{"read_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":1},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259905.058537_842.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:25.058537498Z","data":{"read_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":2},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":3},"status":{"status":"write"}}},{"id":"1_1591259905.064571_843.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:25.064571488Z","data":{"read_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":2},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":3},"status":{"status":"complete"}}},{"id":"1_1591259925.725606_844.1","section":"bucket.instance","name":"gpxthc-27:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.36","timestamp":"2020-06-04T08:38:45.725606301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.729923_861.1","section":"bucket.instance","name":"gpxthc-27:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.36","timestamp":"2020-06-04T08:38:45.729923565Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.732041_862.1","section":"bucket","name":"gpxthc-27","timestamp":"2020-06-04T08:38:45.732041328Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pBr-gACv7tk1mBrJwyLeMiI","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.736204_863.1","section":"bucket","name":"gpxthc-27","timestamp":"2020-06-04T08:38:45.736204921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pBr-gACv7tk1mBrJwyLeMiI","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.764022_864.1","section":"bucket.instance","name":"gpxthc-28:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.37","timestamp":"2020-06-04T08:38:45.764022725Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.768419_865.1","section":"bucket.instance","name":"gpxthc-28:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.37","timestamp":"2020-06-04T08:38:45.768419481Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.770593_866.1","section":"bucket","name":"gpxthc-28","timestamp":"2020-06-04T08:38:45.770593129Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YbhL-HGVrfTpz3aCSgkZ-1i","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.775197_867.1","section":"bucket","name":"gpxthc-28","timestamp":"2020-06-04T08:38:45.775197377Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YbhL-HGVrfTpz3aCSgkZ-1i","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.820803_868.1","section":"bucket.instance","name":"gpxthc-29:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.38","timestamp":"2020-06-04T08:38:45.820803858Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.825390_869.1","section":"bucket.instance","name":"gpxthc-29:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.38","timestamp":"2020-06-04T08:38:45.825390920Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.827444_870.1","section":"bucket","name":"gpxthc-29","timestamp":"2020-06-04T08:38:45.827444882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HJvkKg7YYuNuo_RQ2C1dEVG","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.831901_871.1","section":"bucket","name":"gpxthc-29","timestamp":"2020-06-04T08:38:45.831901352Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HJvkKg7YYuNuo_RQ2C1dEVG","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.856470_872.1","section":"bucket.instance","name":"gpxthc-27:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.36","timestamp":"2020-06-04T08:38:45.856470724Z","data":{"read_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":1},"write_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":2},"status":{"status":"write"}}},{"id":"1_1591259925.861409_873.1","section":"bucket.instance","name":"gpxthc-27:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.36","timestamp":"2020-06-04T08:38:45.861409642Z","data":{"read_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":1},"write_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259925.913880_874.1","section":"bucket.instance","name":"gpxthc-28:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.37","timestamp":"2020-06-04T08:38:45.913880548Z","data":{"read_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":1},"write_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":2},"status":{"status":"write"}}},{"id":"1_1591259925.924621_875.1","section":"bucket.instance","name":"gpxthc-28:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.37","timestamp":"2020-06-04T08:38:45.924621260Z","data":{"read_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":1},"write_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259925.983236_876.1","section":"bucket.instance","name":"gpxthc-29:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.38","timestamp":"2020-06-04T08:38:45.983236325Z","data":{"read_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":1},"write_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":2},"status":{"status":"write"}}},{"id":"1_1591259925.996695_877.1","section":"bucket.instance","name":"gpxthc-29:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.38","timestamp":"2020-06-04T08:38:45.996695629Z","data":{"read_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":1},"write_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259926.077449_878.1","section":"bucket.instance","name":"gpxthc-30:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.39","timestamp":"2020-06-04T08:38:46.077449382Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.083000_879.1","section":"bucket.instance","name":"gpxthc-30:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.39","timestamp":"2020-06-04T08:38:46.083000564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.085045_880.1","section":"bucket","name":"gpxthc-30","timestamp":"2020-06-04T08:38:46.085045274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jthV6t1d3f5V-3IMp3Qs5U2","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.088922_881.1","section":"bucket","name":"gpxthc-30","timestamp":"2020-06-04T08:38:46.088922963Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jthV6t1d3f5V-3IMp3Qs5U2","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.117432_883.1","section":"bucket.instance","name":"gpxthc-31:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.40","timestamp":"2020-06-04T08:38:46.117432535Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.121915_884.1","section":"bucket.instance","name":"gpxthc-31:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.40","timestamp":"2020-06-04T08:38:46.121915138Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.124124_885.1","section":"bucket","name":"gpxthc-31","timestamp":"2020-06-04T08:38:46.124124794Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6ku9WQrR5WvQ3xnFGpcCP0M","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.129078_886.1","section":"bucket","name":"gpxthc-31","timestamp":"2020-06-04T08:38:46.129078803Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6ku9WQrR5WvQ3xnFGpcCP0M","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.175739_887.1","section":"bucket.instance","name":"gpxthc-32:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.41","timestamp":"2020-06-04T08:38:46.175739838Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.182982_888.1","section":"bucket.instance","name":"gpxthc-32:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.41","timestamp":"2020-06-04T08:38:46.182982979Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.186641_889.1","section":"bucket","name":"gpxthc-32","timestamp":"2020-06-04T08:38:46.186641231Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RUtCJbRmBTaCDOrE5uNNCNT","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.193307_890.1","section":"bucket","name":"gpxthc-32","timestamp":"2020-06-04T08:38:46.193307183Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RUtCJbRmBTaCDOrE5uNNCNT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.371575_892.1","section":"bucket.instance","name":"gpxthc-30:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.39","timestamp":"2020-06-04T08:38:46.371575852Z","data":{"read_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":1},"write_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":2},"status":{"status":"write"}}},{"id":"1_1591259926.377365_893.1","section":"bucket.instance","name":"gpxthc-30:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.39","timestamp":"2020-06-04T08:38:46.377365569Z","data":{"read_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":1},"write_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259926.431027_896.1","section":"bucket.instance","name":"gpxthc-31:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.40","timestamp":"2020-06-04T08:38:46.431027842Z","data":{"read_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":1},"write_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":2},"status":{"status":"write"}}},{"id":"1_1591259926.442905_897.1","section":"bucket.instance","name":"gpxthc-31:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.40","timestamp":"2020-06-04T08:38:46.442905741Z","data":{"read_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":1},"write_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259926.503230_898.1","section":"bucket.instance","name":"gpxthc-32:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.41","timestamp":"2020-06-04T08:38:46.503230913Z","data":{"read_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":1},"write_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":2},"status":{"status":"write"}}},{"id":"1_1591259926.513386_899.1","section":"bucket.instance","name":"gpxthc-32:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.41","timestamp":"2020-06-04T08:38:46.513386815Z","data":{"read_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":1},"write_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259926.597611_900.1","section":"bucket.instance","name":"gpxthc-33:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.42","timestamp":"2020-06-04T08:38:46.597611762Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.604278_901.1","section":"bucket.instance","name":"gpxthc-33:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.42","timestamp":"2020-06-04T08:38:46.604278594Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.611816_902.1","section":"bucket","name":"gpxthc-33","timestamp":"2020-06-04T08:38:46.611816568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TCD6G-8d3BbB8YgWl8O3Nwk","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.616028_903.1","section":"bucket","name":"gpxthc-33","timestamp":"2020-06-04T08:38:46.616028597Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TCD6G-8d3BbB8YgWl8O3Nwk","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.637171_904.1","section":"bucket.instance","name":"gpxthc-34:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.43","timestamp":"2020-06-04T08:38:46.637171077Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.641863_905.1","section":"bucket.instance","name":"gpxthc-34:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.43","timestamp":"2020-06-04T08:38:46.641863656Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.644434_906.1","section":"bucket","name":"gpxthc-34","timestamp":"2020-06-04T08:38:46.644434253Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P-m8zGy2fOnlsKo8No6u1Cq","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.652520_907.1","section":"bucket","name":"gpxthc-34","timestamp":"2020-06-04T08:38:46.652520435Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P-m8zGy2fOnlsKo8No6u1Cq","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.695989_908.1","section":"bucket.instance","name":"gpxthc-35:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.44","timestamp":"2020-06-04T08:38:46.695989087Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.700896_909.1","section":"bucket.instance","name":"gpxthc-35:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.44","timestamp":"2020-06-04T08:38:46.700896824Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.703158_910.1","section":"bucket","name":"gpxthc-35","timestamp":"2020-06-04T08:38:46.703158174Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gEvrMnPJFZrfyPUGCekqix_","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.707672_911.1","section":"bucket","name":"gpxthc-35","timestamp":"2020-06-04T08:38:46.707672554Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gEvrMnPJFZrfyPUGCekqix_","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.732054_912.1","section":"bucket.instance","name":"gpxthc-33:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.42","timestamp":"2020-06-04T08:38:46.732054711Z","data":{"read_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":1},"write_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":2},"status":{"status":"write"}}},{"id":"1_1591259926.737279_913.1","section":"bucket.instance","name":"gpxthc-33:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.42","timestamp":"2020-06-04T08:38:46.737279135Z","data":{"read_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":1},"write_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259927.790025_914.1","section":"bucket.instance","name":"gpxthc-34:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.43","timestamp":"2020-06-04T08:38:47.790025773Z","data":{"read_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":1},"write_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":2},"status":{"status":"write"}}},{"id":"1_1591259927.802447_915.1","section":"bucket.instance","name":"gpxthc-34:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.43","timestamp":"2020-06-04T08:38:47.802447374Z","data":{"read_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":1},"write_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259928.861627_916.1","section":"bucket.instance","name":"gpxthc-35:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.44","timestamp":"2020-06-04T08:38:48.861627553Z","data":{"read_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":1},"write_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":2},"status":{"status":"write"}}},{"id":"1_1591259928.873592_917.1","section":"bucket.instance","name":"gpxthc-35:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.44","timestamp":"2020-06-04T08:38:48.873592348Z","data":{"read_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":1},"write_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259928.954520_918.1","section":"bucket.instance","name":"gpxthc-36:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.45","timestamp":"2020-06-04T08:38:48.954520125Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_11-lNmUdWJkycuWy12Rkais","ver":1},"status":{"status":"write"}}},{"id":"1_1591259928.958725_919.1","section":"bucket.instance","name":"gpxthc-36:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.45","timestamp":"2020-06-04T08:38:48.958725871Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_11-lNmUdWJkycuWy12Rkais","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259928.960607_920.1","section":"bucket","name":"gpxthc-36","timestamp":"2020-06-04T08:38:48.960607582Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2lIEwet-R7Pmma_XC0XfJfn","ver":1},"status":{"status":"write"}}},{"id":"1_1591259928.964925_921.1","section":"bucket","name":"gpxthc-36","timestamp":"2020-06-04T08:38:48.964925190Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2lIEwet-R7Pmma_XC0XfJfn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259929.002482_922.1","section":"bucket.instance","name":"gpxthc-37:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.46","timestamp":"2020-06-04T08:38:49.002482783Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V-CEvpx3s4VyGthwNmj1yAU","ver":1},"status":{"status":"write"}}},{"id":"1_1591259929.007285_923.1","section":"bucket.instance","name":"gpxthc-37:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.46","timestamp":"2020-06-04T08:38:49.007285334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V-CEvpx3s4VyGthwNmj1yAU","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259929.009398_924.1","section":"bucket","name":"gpxthc-37","timestamp":"2020-06-04T08:38:49.009398457Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ek_-Wa_Uyu-XnjfASqAxdXy","ver":1},"status":{"status":"write"}}},{"id":"1_1591259929.013412_925.1","section":"bucket","name":"gpxthc-37","timestamp":"2020-06-04T08:38:49.013412886Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ek_-Wa_Uyu-XnjfASqAxdXy","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259929.052573_926.1","section":"bucket.instance","name":"gpxthc-38:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.47","timestamp":"2020-06-04T08:38:49.052573944Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N6EHuJJFZ7_2dQ30ZsrYxhQ","ver":1},"status":{"status":"write"}}},{"id":"1_1591259929.056846_927.1","section":"bucket.instance","name":"gpxthc-38:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.47","timestamp":"2020-06-04T08:38:49.056846763Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N6EHuJJFZ7_2dQ30ZsrYxhQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259929.059064_928.1","section":"bucket","name":"gpxthc-38","timestamp":"2020-06-04T08:38:49.059064075Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-zv4Cksv-zkyzdIqRjsaDb2","ver":1},"status":{"status":"write"}}},{"id":"1_1591259929.063266_929.1","section":"bucket","name":"gpxthc-38","timestamp":"2020-06-04T08:38:49.063266714Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-zv4Cksv-zkyzdIqRjsaDb2","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.609274_930.1","section":"bucket.instance","name":"gpxthc-39:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.48","timestamp":"2020-06-04T08:38:50.609274737Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VhICsf9zqk7f9jpddSNIqkP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.613351_932.1","section":"bucket.instance","name":"gpxthc-39:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.48","timestamp":"2020-06-04T08:38:50.613351600Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VhICsf9zqk7f9jpddSNIqkP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.615477_933.1","section":"bucket","name":"gpxthc-39","timestamp":"2020-06-04T08:38:50.615477264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v902jpCGpnCkP2gKKzOXdqt","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.619346_934.1","section":"bucket","name":"gpxthc-39","timestamp":"2020-06-04T08:38:50.619346741Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v902jpCGpnCkP2gKKzOXdqt","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.653610_935.1","section":"bucket.instance","name":"gpxthc-40:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.49","timestamp":"2020-06-04T08:38:50.653610015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pbAYVMP2sAf_BuaE5BYBTkr","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.658862_936.1","section":"bucket.instance","name":"gpxthc-40:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.49","timestamp":"2020-06-04T08:38:50.658862411Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pbAYVMP2sAf_BuaE5BYBTkr","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.661191_937.1","section":"bucket","name":"gpxthc-40","timestamp":"2020-06-04T08:38:50.661191135Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4cw4XeidBA2TPzMfHBQhnsn","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.665356_938.1","section":"bucket","name":"gpxthc-40","timestamp":"2020-06-04T08:38:50.665356122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4cw4XeidBA2TPzMfHBQhnsn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.711577_939.1","section":"bucket.instance","name":"gpxthc-41:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.50","timestamp":"2020-06-04T08:38:50.711577903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zFKqF0DR8qyD3-Tr3FbcjW1","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.719322_940.1","section":"bucket.instance","name":"gpxthc-41:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.50","timestamp":"2020-06-04T08:38:50.719322092Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zFKqF0DR8qyD3-Tr3FbcjW1","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.721242_941.1","section":"bucket","name":"gpxthc-41","timestamp":"2020-06-04T08:38:50.721242218Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n_mX6j5IFhQpmY51w3jLeA3","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.725192_942.1","section":"bucket","name":"gpxthc-41","timestamp":"2020-06-04T08:38:50.725192405Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n_mX6j5IFhQpmY51w3jLeA3","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259341.737046_34.1","section":"bucket.instance","name":"gpxthc-1:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.1","timestamp":"2020-06-04T08:29:01.737046909Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-VXJYi2_CLvrsqw6qc6DCr5","ver":1},"status":{"status":"write"}}},{"id":"1_1591259341.740440_35.1","section":"bucket.instance","name":"gpxthc-1:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.1","timestamp":"2020-06-04T08:29:01.740440081Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-VXJYi2_CLvrsqw6qc6DCr5","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259341.742105_36.1","section":"bucket","name":"gpxthc-1","timestamp":"2020-06-04T08:29:01.742105318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6GkvMPsGVWRJghpjcfPP5td","ver":1},"status":{"status":"write"}}},{"id":"1_1591259341.745009_37.1","section":"bucket","name":"gpxthc-1","timestamp":"2020-06-04T08:29:01.745009929Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6GkvMPsGVWRJghpjcfPP5td","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259341.768025_38.1","section":"bucket.instance","name":"gpxthc-2:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.2","timestamp":"2020-06-04T08:29:01.768025171Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XvNT082cvaVbqbr-2ZESWeT","ver":1},"status":{"status":"write"}}},{"id":"1_1591259341.771975_39.1","section":"bucket.instance","name":"gpxthc-2:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.2","timestamp":"2020-06-04T08:29:01.771975782Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XvNT082cvaVbqbr-2ZESWeT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259341.773834_40.1","section":"bucket","name":"gpxthc-2","timestamp":"2020-06-04T08:29:01.773834414Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AQGxlKt5b4yEuqgCWbWM_je","ver":1},"status":{"status":"write"}}},{"id":"1_1591259341.777323_41.1","section":"bucket","name":"gpxthc-2","timestamp":"2020-06-04T08:29:01.777323422Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AQGxlKt5b4yEuqgCWbWM_je","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259345.200120_42.1","section":"bucket.instance","name":"gpxthc-3:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.3","timestamp":"2020-06-04T08:29:05.200120296Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r--8CKmcPrYbCzDhO-F06sP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259345.203880_43.1","section":"bucket.instance","name":"gpxthc-3:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.3","timestamp":"2020-06-04T08:29:05.203880111Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r--8CKmcPrYbCzDhO-F06sP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259345.205531_44.1","section":"bucket","name":"gpxthc-3","timestamp":"2020-06-04T08:29:05.205531158Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bgP8ilt4JXWshxEQEKDTOxL","ver":1},"status":{"status":"write"}}},{"id":"1_1591259345.209069_45.1","section":"bucket","name":"gpxthc-3","timestamp":"2020-06-04T08:29:05.209069471Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bgP8ilt4JXWshxEQEKDTOxL","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.383145_46.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.4","timestamp":"2020-06-04T08:29:06.383145515Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HbOqQVlA66qYh_U-6gTxraM","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.388144_47.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.4","timestamp":"2020-06-04T08:29:06.388144053Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HbOqQVlA66qYh_U-6gTxraM","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.390123_48.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:06.390123332Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ThXePFlF36ZdSx-rFItfhQ8","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.394528_49.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:06.394528126Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ThXePFlF36ZdSx-rFItfhQ8","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.415142_50.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.5","timestamp":"2020-06-04T08:29:06.415142498Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FgaMM_80CprME0J7uTE6atP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.419201_51.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.5","timestamp":"2020-06-04T08:29:06.419201114Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FgaMM_80CprME0J7uTE6atP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.421086_52.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:06.421086802Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Nj0SyzQoWhfLBcf7quWXETq","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.424778_53.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:06.424778499Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Nj0SyzQoWhfLBcf7quWXETq","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.463965_54.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.6","timestamp":"2020-06-04T08:29:06.463965385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-cllnOXQdxzDqATQ1dbWkmx","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.467779_55.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.6","timestamp":"2020-06-04T08:29:06.467779786Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-cllnOXQdxzDqATQ1dbWkmx","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.469502_56.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:06.469502201Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uaSCzOL2DVEOK-odgsLCwFs","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.473034_57.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:06.473034822Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uaSCzOL2DVEOK-odgsLCwFs","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.614173_58.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.7","timestamp":"2020-06-04T08:29:07.614173009Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QW7H8a3CUThQ_bRH4yAqRKv","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.618181_59.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.7","timestamp":"2020-06-04T08:29:07.618181728Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QW7H8a3CUThQ_bRH4yAqRKv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.619881_60.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:07.619881218Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5sgvOjv3iB27roqBQ7cUZbh","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.623348_61.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:07.623348297Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5sgvOjv3iB27roqBQ7cUZbh","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259347.647460_62.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.8","timestamp":"2020-06-04T08:29:07.647460278Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eGGXzzZG1A9GEn7uWckVfr-","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.651908_63.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.8","timestamp":"2020-06-04T08:29:07.651908533Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eGGXzzZG1A9GEn7uWckVfr-","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.654187_64.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:07.654187090Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UJzl8Whz1V5QSBu3ohXNupT","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.657537_65.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:07.657537293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UJzl8Whz1V5QSBu3ohXNupT","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259347.686764_66.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.9","timestamp":"2020-06-04T08:29:07.686764572Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vZquMGd6hFx42B6ADczwHim","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.691403_67.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.9","timestamp":"2020-06-04T08:29:07.691403042Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vZquMGd6hFx42B6ADczwHim","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.693513_68.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:07.693513613Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UOipH4LH_UUuQBoXOTlPqdc","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.698070_69.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:07.698070773Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UOipH4LH_UUuQBoXOTlPqdc","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259347.733629_70.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.10","timestamp":"2020-06-04T08:29:07.733629268Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4NNS9aPc6N_RBBTATY5XJN5","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.737752_71.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.10","timestamp":"2020-06-04T08:29:07.737752160Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4NNS9aPc6N_RBBTATY5XJN5","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.739454_72.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:07.739454023Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9rrBHViPIaHB1oKA4-fy2a7","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.742970_73.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:07.742970942Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9rrBHViPIaHB1oKA4-fy2a7","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259347.792004_74.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.11","timestamp":"2020-06-04T08:29:07.792004579Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R8wjaedoiarBcdJxSdsNG9i","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.797167_75.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.11","timestamp":"2020-06-04T08:29:07.797167708Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R8wjaedoiarBcdJxSdsNG9i","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.798799_76.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:07.798799372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFqu-tIqXwMndR9GZsMEdUg","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.801827_77.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:07.801827701Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFqu-tIqXwMndR9GZsMEdUg","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259347.895642_78.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.12","timestamp":"2020-06-04T08:29:07.895642957Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_813CHCsdbVJhG-8Friwea0Q","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.900475_79.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.12","timestamp":"2020-06-04T08:29:07.900475349Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_813CHCsdbVJhG-8Friwea0Q","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.903751_80.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:07.903751639Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4jTRpPwqpn_U-wRoBFCv-5t","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.907394_81.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:07.907394318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4jTRpPwqpn_U-wRoBFCv-5t","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259348.003913_82.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.13","timestamp":"2020-06-04T08:29:08.003913702Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vjl-KiOIIS-mHj4Knkvyvbn","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.008410_83.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.13","timestamp":"2020-06-04T08:29:08.008410230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vjl-KiOIIS-mHj4Knkvyvbn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259348.010405_84.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:08.010405065Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YXdPZK993bwZgNRYKqmPKbg","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.013902_85.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:08.013902468Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YXdPZK993bwZgNRYKqmPKbg","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259348.061824_86.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.14","timestamp":"2020-06-04T08:29:08.061824583Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GmPkZfz9onYDYux9NvroVpb","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.066453_87.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.14","timestamp":"2020-06-04T08:29:08.066453149Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GmPkZfz9onYDYux9NvroVpb","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259348.068278_88.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:08.068278859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n5UFL0MW6wtQDqqBX5p9xOv","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.071562_89.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:08.071562898Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n5UFL0MW6wtQDqqBX5p9xOv","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259348.164678_90.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.15","timestamp":"2020-06-04T08:29:08.164678779Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wnNQvYf5A6zQTv_L1L16Vj_","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.169209_91.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.15","timestamp":"2020-06-04T08:29:08.169209509Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wnNQvYf5A6zQTv_L1L16Vj_","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259348.171482_92.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:08.171482220Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HuczynHZJw9zrk53biUsBon","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.175359_93.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:08.175359125Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HuczynHZJw9zrk53biUsBon","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259349.444497_94.1","section":"bucket.instance","name":"gpxthc-7:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.16","timestamp":"2020-06-04T08:29:09.444497949Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0hH3KdNQ7Ror_-PbjMomrOK","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.449423_95.1","section":"bucket.instance","name":"gpxthc-7:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.16","timestamp":"2020-06-04T08:29:09.449423359Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0hH3KdNQ7Ror_-PbjMomrOK","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259349.451676_96.1","section":"bucket","name":"gpxthc-7","timestamp":"2020-06-04T08:29:09.451676822Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.455775_97.1","section":"bucket","name":"gpxthc-7","timestamp":"2020-06-04T08:29:09.455775359Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259349.478408_98.1","section":"bucket.instance","name":"gpxthc-8:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.17","timestamp":"2020-06-04T08:29:09.478408677Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3tdK5SOwUxP7sixtd5Iiusu","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.483441_99.1","section":"bucket.instance","name":"gpxthc-8:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.17","timestamp":"2020-06-04T08:29:09.483441043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3tdK5SOwUxP7sixtd5Iiusu","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259349.485210_100.1","section":"bucket","name":"gpxthc-8","timestamp":"2020-06-04T08:29:09.485210362Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.489008_101.1","section":"bucket","name":"gpxthc-8","timestamp":"2020-06-04T08:29:09.489008069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259349.542500_102.1","section":"bucket.instance","name":"gpxthc-9:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.18","timestamp":"2020-06-04T08:29:09.542500556Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZM4c0G2TMLKguEZQHJhJ_U1","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.547391_103.1","section":"bucket.instance","name":"gpxthc-9:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.18","timestamp":"2020-06-04T08:29:09.547391664Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZM4c0G2TMLKguEZQHJhJ_U1","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259349.549662_104.1","section":"bucket","name":"gpxthc-9","timestamp":"2020-06-04T08:29:09.549662472Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.557874_105.1","section":"bucket","name":"gpxthc-9","timestamp":"2020-06-04T08:29:09.557874857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259350.809493_106.1","section":"bucket","name":"gpxthc-7","timestamp":"2020-06-04T08:29:10.809493171Z","data":{"read_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":1},"write_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":2},"status":{"status":"remove"}}},{"id":"1_1591259350.815979_107.1","section":"bucket","name":"gpxthc-7","timestamp":"2020-06-04T08:29:10.815979805Z","data":{"read_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":1},"write_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259350.882584_108.1","section":"bucket","name":"gpxthc-8","timestamp":"2020-06-04T08:29:10.882584350Z","data":{"read_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":1},"write_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":2},"status":{"status":"remove"}}},{"id":"1_1591259350.890059_109.1","section":"bucket","name":"gpxthc-8","timestamp":"2020-06-04T08:29:10.890059979Z","data":{"read_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":1},"write_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259350.978944_110.1","section":"bucket","name":"gpxthc-9","timestamp":"2020-06-04T08:29:10.978944722Z","data":{"read_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":1},"write_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":2},"status":{"status":"remove"}}},{"id":"1_1591259350.985454_111.1","section":"bucket","name":"gpxthc-9","timestamp":"2020-06-04T08:29:10.985454315Z","data":{"read_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":1},"write_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":2},"status":{"status":"complete"}}}] 2020-06-04T08:41:13.465 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '838ed6b2-fa2f-4917-a5c1-6c82bb405cda', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:13.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '838ed6b2-fa2f-4917-a5c1-6c82bb405cda', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:13.466 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 838ed6b2-fa2f-4917-a5c1-6c82bb405cda --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:13.510 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:13.510 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:13.780 INFO:teuthology.orchestra.run.smithi060.stdout:[{"id":"1_1591259219.822557_39.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-06-04T08:26:59.822557133Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UKgIH850zXzbqniMBbCDHLr","ver":1},"status":{"status":"write"}}},{"id":"1_1591259219.830770_41.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-06-04T08:26:59.830770292Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UKgIH850zXzbqniMBbCDHLr","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.183102_42.1","section":"bucket.instance","name":"gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19","timestamp":"2020-06-04T08:29:12.183102477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VfH48PXmsZggFiAsqBANxvt","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.187271_50.1","section":"bucket.instance","name":"gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19","timestamp":"2020-06-04T08:29:12.187271885Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VfH48PXmsZggFiAsqBANxvt","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.189031_51.1","section":"bucket","name":"gpxthc-10","timestamp":"2020-06-04T08:29:12.189031257Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_h8WUqJo-QErjBMZIQdgNCtn","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.192886_52.1","section":"bucket","name":"gpxthc-10","timestamp":"2020-06-04T08:29:12.192886901Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_h8WUqJo-QErjBMZIQdgNCtn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.214300_53.1","section":"bucket.instance","name":"gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20","timestamp":"2020-06-04T08:29:12.214300618Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OITM9jIqMd3jh6APsdWU31A","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.219348_54.1","section":"bucket.instance","name":"gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20","timestamp":"2020-06-04T08:29:12.219348485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OITM9jIqMd3jh6APsdWU31A","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.221278_55.1","section":"bucket","name":"gpxthc-11","timestamp":"2020-06-04T08:29:12.221278412Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-nUpFbAACCD5rYXC7DcxUQR","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.225194_56.1","section":"bucket","name":"gpxthc-11","timestamp":"2020-06-04T08:29:12.225194355Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-nUpFbAACCD5rYXC7DcxUQR","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.268612_57.1","section":"bucket.instance","name":"gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21","timestamp":"2020-06-04T08:29:12.268612619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zd8ii08n-ZihNb68aNArIKM","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.273166_58.1","section":"bucket.instance","name":"gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21","timestamp":"2020-06-04T08:29:12.273166922Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zd8ii08n-ZihNb68aNArIKM","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.275116_59.1","section":"bucket","name":"gpxthc-12","timestamp":"2020-06-04T08:29:12.275116572Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AleC4XiO8GV6Mta70FMehuN","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.279189_60.1","section":"bucket","name":"gpxthc-12","timestamp":"2020-06-04T08:29:12.279189055Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AleC4XiO8GV6Mta70FMehuN","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.619667_61.1","section":"bucket.instance","name":"gpxthc-13:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.22","timestamp":"2020-06-04T08:29:31.619667228Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G4zU5SP7vzOavY4UDRy0kVT","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.624339_68.1","section":"bucket.instance","name":"gpxthc-13:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.22","timestamp":"2020-06-04T08:29:31.624339358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G4zU5SP7vzOavY4UDRy0kVT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.626643_69.1","section":"bucket","name":"gpxthc-13","timestamp":"2020-06-04T08:29:31.626643743Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VUOAgr-b5NnlCfY5DmWhaqq","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.630935_70.1","section":"bucket","name":"gpxthc-13","timestamp":"2020-06-04T08:29:31.630935737Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VUOAgr-b5NnlCfY5DmWhaqq","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.653316_71.1","section":"bucket.instance","name":"gpxthc-14:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.23","timestamp":"2020-06-04T08:29:31.653316611Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YhqpqT7kUaddbo1-1KfMhvv","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.658400_72.1","section":"bucket.instance","name":"gpxthc-14:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.23","timestamp":"2020-06-04T08:29:31.658400301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YhqpqT7kUaddbo1-1KfMhvv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.660536_73.1","section":"bucket","name":"gpxthc-14","timestamp":"2020-06-04T08:29:31.660536384Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Br3NXT9kNBdoMbsZqQdcMrv","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.664400_74.1","section":"bucket","name":"gpxthc-14","timestamp":"2020-06-04T08:29:31.664400106Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Br3NXT9kNBdoMbsZqQdcMrv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.712373_75.1","section":"bucket.instance","name":"gpxthc-15:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.24","timestamp":"2020-06-04T08:29:31.712373958Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_frk3BKkQ_Qa4KdotA_oTnm0","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.716427_76.1","section":"bucket.instance","name":"gpxthc-15:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.24","timestamp":"2020-06-04T08:29:31.716427746Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_frk3BKkQ_Qa4KdotA_oTnm0","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.719782_77.1","section":"bucket","name":"gpxthc-15","timestamp":"2020-06-04T08:29:31.719782440Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CRGlFFvUIxHRKMSeAVvS4kF","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.724712_78.1","section":"bucket","name":"gpxthc-15","timestamp":"2020-06-04T08:29:31.724712205Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CRGlFFvUIxHRKMSeAVvS4kF","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.428231_79.1","section":"bucket.instance","name":"gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25","timestamp":"2020-06-04T08:30:16.428231743Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.433809_94.1","section":"bucket.instance","name":"gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25","timestamp":"2020-06-04T08:30:16.433809103Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.435876_95.1","section":"bucket","name":"gpxthc-16","timestamp":"2020-06-04T08:30:16.435876487Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4nNLQIaMSuVxPP12bYBinmN","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.440082_96.1","section":"bucket","name":"gpxthc-16","timestamp":"2020-06-04T08:30:16.440082082Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4nNLQIaMSuVxPP12bYBinmN","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.465814_97.1","section":"bucket.instance","name":"gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26","timestamp":"2020-06-04T08:30:16.465814342Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.470277_98.1","section":"bucket.instance","name":"gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26","timestamp":"2020-06-04T08:30:16.470277700Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.472163_99.1","section":"bucket","name":"gpxthc-17","timestamp":"2020-06-04T08:30:16.472163913Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jIqc-jlMoWLqci8_G-SuMb2","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.476487_100.1","section":"bucket","name":"gpxthc-17","timestamp":"2020-06-04T08:30:16.476487448Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jIqc-jlMoWLqci8_G-SuMb2","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.517171_101.1","section":"bucket.instance","name":"gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27","timestamp":"2020-06-04T08:30:16.517171806Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.522567_102.1","section":"bucket.instance","name":"gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27","timestamp":"2020-06-04T08:30:16.522567636Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.524592_103.1","section":"bucket","name":"gpxthc-18","timestamp":"2020-06-04T08:30:16.524592149Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T8PKYfLd2MscBs2JJLeNmbu","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.529893_104.1","section":"bucket","name":"gpxthc-18","timestamp":"2020-06-04T08:30:16.529893092Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T8PKYfLd2MscBs2JJLeNmbu","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.564959_105.1","section":"bucket.instance","name":"gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25","timestamp":"2020-06-04T08:30:16.564959314Z","data":{"read_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":1},"write_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":2},"status":{"status":"write"}}},{"id":"1_1591259416.569413_106.1","section":"bucket.instance","name":"gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25","timestamp":"2020-06-04T08:30:16.569413363Z","data":{"read_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":1},"write_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259416.573339_107.1","section":"bucket.instance","name":"gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26","timestamp":"2020-06-04T08:30:16.573339571Z","data":{"read_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":1},"write_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":2},"status":{"status":"write"}}},{"id":"1_1591259416.578468_108.1","section":"bucket.instance","name":"gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26","timestamp":"2020-06-04T08:30:16.578468486Z","data":{"read_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":1},"write_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259416.593653_109.1","section":"bucket.instance","name":"gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27","timestamp":"2020-06-04T08:30:16.593653914Z","data":{"read_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":1},"write_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":2},"status":{"status":"write"}}},{"id":"1_1591259416.599911_110.1","section":"bucket.instance","name":"gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27","timestamp":"2020-06-04T08:30:16.599911549Z","data":{"read_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":1},"write_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259628.323783_111.1","section":"bucket.instance","name":"gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28","timestamp":"2020-06-04T08:33:48.323783159Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":1},"status":{"status":"write"}}},{"id":"1_1591259628.328728_148.1","section":"bucket.instance","name":"gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28","timestamp":"2020-06-04T08:33:48.328728853Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259628.330710_149.1","section":"bucket","name":"gpxthc-19","timestamp":"2020-06-04T08:33:48.330710900Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_93QBHlBvEVUFT9b_wxOIMnq","ver":1},"status":{"status":"write"}}},{"id":"1_1591259628.334787_150.1","section":"bucket","name":"gpxthc-19","timestamp":"2020-06-04T08:33:48.334787040Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_93QBHlBvEVUFT9b_wxOIMnq","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259628.341176_152.1","section":"bucket.instance","name":"gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28","timestamp":"2020-06-04T08:33:48.341176479Z","data":{"read_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":1},"write_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":2},"status":{"status":"write"}}},{"id":"1_1591259628.347410_153.1","section":"bucket.instance","name":"gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28","timestamp":"2020-06-04T08:33:48.347410493Z","data":{"read_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":1},"write_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259767.841895_739.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:36:07.841895512Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":1},"status":{"status":"write"}}},{"id":"1_1591259767.846376_745.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:36:07.846376164Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259767.848773_746.1","section":"bucket","name":"gpxthc-20","timestamp":"2020-06-04T08:36:07.848773543Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ErTLIwy11AZgBe_n6j9EOvj","ver":1},"status":{"status":"write"}}},{"id":"1_1591259767.852808_747.1","section":"bucket","name":"gpxthc-20","timestamp":"2020-06-04T08:36:07.852808654Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ErTLIwy11AZgBe_n6j9EOvj","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259774.352869_748.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:36:14.352869429Z","data":{"read_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":1},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":2},"status":{"status":"write"}}},{"id":"1_1591259774.363917_755.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:36:14.363917038Z","data":{"read_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":1},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259833.992686_770.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:37:13.992686192Z","data":{"read_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":2},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":3},"status":{"status":"write"}}},{"id":"1_1591259834.004633_772.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:37:14.004633298Z","data":{"read_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":2},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":3},"status":{"status":"complete"}}},{"id":"1_1591259877.840447_773.1","section":"bucket.instance","name":"gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30","timestamp":"2020-06-04T08:37:57.840447936Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.844931_787.1","section":"bucket.instance","name":"gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30","timestamp":"2020-06-04T08:37:57.844931870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.847150_788.1","section":"bucket","name":"gpxthc-21","timestamp":"2020-06-04T08:37:57.847150642Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N959Mvlm6EfOalZxV2Ys2gP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.851202_789.1","section":"bucket","name":"gpxthc-21","timestamp":"2020-06-04T08:37:57.851202385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N959Mvlm6EfOalZxV2Ys2gP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.876704_790.1","section":"bucket.instance","name":"gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31","timestamp":"2020-06-04T08:37:57.876704072Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.881398_791.1","section":"bucket.instance","name":"gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31","timestamp":"2020-06-04T08:37:57.881398442Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.883373_792.1","section":"bucket","name":"gpxthc-22","timestamp":"2020-06-04T08:37:57.883373243Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dEQaV8gUPyD0QSgk_KOV4um","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.887399_793.1","section":"bucket","name":"gpxthc-22","timestamp":"2020-06-04T08:37:57.887399447Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dEQaV8gUPyD0QSgk_KOV4um","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.931319_794.1","section":"bucket.instance","name":"gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32","timestamp":"2020-06-04T08:37:57.931319262Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.936911_795.1","section":"bucket.instance","name":"gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32","timestamp":"2020-06-04T08:37:57.936911048Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.939469_796.1","section":"bucket","name":"gpxthc-23","timestamp":"2020-06-04T08:37:57.939469678Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c3fe5iwyxyrYzoUjllCpsvU","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.943713_797.1","section":"bucket","name":"gpxthc-23","timestamp":"2020-06-04T08:37:57.943713180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c3fe5iwyxyrYzoUjllCpsvU","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.969944_798.1","section":"bucket.instance","name":"gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30","timestamp":"2020-06-04T08:37:57.969944576Z","data":{"read_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":1},"write_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":2},"status":{"status":"write"}}},{"id":"1_1591259877.980272_799.1","section":"bucket.instance","name":"gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30","timestamp":"2020-06-04T08:37:57.980272398Z","data":{"read_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":1},"write_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259877.984533_800.1","section":"bucket.instance","name":"gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31","timestamp":"2020-06-04T08:37:57.984533518Z","data":{"read_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":1},"write_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":2},"status":{"status":"write"}}},{"id":"1_1591259877.989050_801.1","section":"bucket.instance","name":"gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31","timestamp":"2020-06-04T08:37:57.989050456Z","data":{"read_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":1},"write_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259878.000419_802.1","section":"bucket.instance","name":"gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32","timestamp":"2020-06-04T08:37:58.000419669Z","data":{"read_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":1},"write_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":2},"status":{"status":"write"}}},{"id":"1_1591259878.006196_803.1","section":"bucket.instance","name":"gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32","timestamp":"2020-06-04T08:37:58.006196070Z","data":{"read_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":1},"write_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259904.804937_804.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.804937045Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.810097_821.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.810097096Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.812274_822.1","section":"bucket","name":"gpxthc-24","timestamp":"2020-06-04T08:38:24.812274099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RQ8eJ9g70-tjyJRj0QIilAr","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.816235_823.1","section":"bucket","name":"gpxthc-24","timestamp":"2020-06-04T08:38:24.816235964Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RQ8eJ9g70-tjyJRj0QIilAr","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.846016_824.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:24.846016920Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.850663_825.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:24.850663462Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.852772_826.1","section":"bucket","name":"gpxthc-25","timestamp":"2020-06-04T08:38:24.852772198Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nSe4ph9lkroFUUb_S85OnlC","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.857137_827.1","section":"bucket","name":"gpxthc-25","timestamp":"2020-06-04T08:38:24.857137822Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nSe4ph9lkroFUUb_S85OnlC","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.906938_828.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:24.906938522Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.912058_829.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:24.912058812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.914274_830.1","section":"bucket","name":"gpxthc-26","timestamp":"2020-06-04T08:38:24.914274818Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_edlwuY9ktEE0aOm2248enxI","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.918216_831.1","section":"bucket","name":"gpxthc-26","timestamp":"2020-06-04T08:38:24.918216849Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_edlwuY9ktEE0aOm2248enxI","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.939673_832.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.939673512Z","data":{"read_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":1},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":2},"status":{"status":"write"}}},{"id":"1_1591259904.945900_833.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.945900417Z","data":{"read_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":1},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259904.952407_834.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.952407248Z","data":{"read_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":2},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":3},"status":{"status":"write"}}},{"id":"1_1591259904.959605_835.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.959605042Z","data":{"read_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":2},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":3},"status":{"status":"complete"}}},{"id":"1_1591259904.969679_836.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:24.969679222Z","data":{"read_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":1},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":2},"status":{"status":"write"}}},{"id":"1_1591259904.981104_837.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:24.981104198Z","data":{"read_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":1},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259905.015511_838.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:25.015511493Z","data":{"read_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":2},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":3},"status":{"status":"write"}}},{"id":"1_1591259905.023462_839.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:25.023462062Z","data":{"read_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":2},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":3},"status":{"status":"complete"}}},{"id":"1_1591259905.044506_840.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:25.044506828Z","data":{"read_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":1},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":2},"status":{"status":"write"}}},{"id":"1_1591259905.049929_841.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:25.049929340Z","data":{"read_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":1},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259905.058537_842.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:25.058537498Z","data":{"read_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":2},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":3},"status":{"status":"write"}}},{"id":"1_1591259905.064571_843.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:25.064571488Z","data":{"read_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":2},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":3},"status":{"status":"complete"}}},{"id":"1_1591259925.725606_844.1","section":"bucket.instance","name":"gpxthc-27:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.36","timestamp":"2020-06-04T08:38:45.725606301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.729923_861.1","section":"bucket.instance","name":"gpxthc-27:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.36","timestamp":"2020-06-04T08:38:45.729923565Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.732041_862.1","section":"bucket","name":"gpxthc-27","timestamp":"2020-06-04T08:38:45.732041328Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pBr-gACv7tk1mBrJwyLeMiI","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.736204_863.1","section":"bucket","name":"gpxthc-27","timestamp":"2020-06-04T08:38:45.736204921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pBr-gACv7tk1mBrJwyLeMiI","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.764022_864.1","section":"bucket.instance","name":"gpxthc-28:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.37","timestamp":"2020-06-04T08:38:45.764022725Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.768419_865.1","section":"bucket.instance","name":"gpxthc-28:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.37","timestamp":"2020-06-04T08:38:45.768419481Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.770593_866.1","section":"bucket","name":"gpxthc-28","timestamp":"2020-06-04T08:38:45.770593129Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YbhL-HGVrfTpz3aCSgkZ-1i","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.775197_867.1","section":"bucket","name":"gpxthc-28","timestamp":"2020-06-04T08:38:45.775197377Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YbhL-HGVrfTpz3aCSgkZ-1i","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.820803_868.1","section":"bucket.instance","name":"gpxthc-29:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.38","timestamp":"2020-06-04T08:38:45.820803858Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.825390_869.1","section":"bucket.instance","name":"gpxthc-29:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.38","timestamp":"2020-06-04T08:38:45.825390920Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.827444_870.1","section":"bucket","name":"gpxthc-29","timestamp":"2020-06-04T08:38:45.827444882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HJvkKg7YYuNuo_RQ2C1dEVG","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.831901_871.1","section":"bucket","name":"gpxthc-29","timestamp":"2020-06-04T08:38:45.831901352Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HJvkKg7YYuNuo_RQ2C1dEVG","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.856470_872.1","section":"bucket.instance","name":"gpxthc-27:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.36","timestamp":"2020-06-04T08:38:45.856470724Z","data":{"read_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":1},"write_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":2},"status":{"status":"write"}}},{"id":"1_1591259925.861409_873.1","section":"bucket.instance","name":"gpxthc-27:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.36","timestamp":"2020-06-04T08:38:45.861409642Z","data":{"read_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":1},"write_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259925.913880_874.1","section":"bucket.instance","name":"gpxthc-28:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.37","timestamp":"2020-06-04T08:38:45.913880548Z","data":{"read_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":1},"write_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":2},"status":{"status":"write"}}},{"id":"1_1591259925.924621_875.1","section":"bucket.instance","name":"gpxthc-28:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.37","timestamp":"2020-06-04T08:38:45.924621260Z","data":{"read_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":1},"write_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259925.983236_876.1","section":"bucket.instance","name":"gpxthc-29:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.38","timestamp":"2020-06-04T08:38:45.983236325Z","data":{"read_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":1},"write_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":2},"status":{"status":"write"}}},{"id":"1_1591259925.996695_877.1","section":"bucket.instance","name":"gpxthc-29:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.38","timestamp":"2020-06-04T08:38:45.996695629Z","data":{"read_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":1},"write_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259926.077449_878.1","section":"bucket.instance","name":"gpxthc-30:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.39","timestamp":"2020-06-04T08:38:46.077449382Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.083000_879.1","section":"bucket.instance","name":"gpxthc-30:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.39","timestamp":"2020-06-04T08:38:46.083000564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.085045_880.1","section":"bucket","name":"gpxthc-30","timestamp":"2020-06-04T08:38:46.085045274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jthV6t1d3f5V-3IMp3Qs5U2","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.088922_881.1","section":"bucket","name":"gpxthc-30","timestamp":"2020-06-04T08:38:46.088922963Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jthV6t1d3f5V-3IMp3Qs5U2","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.117432_883.1","section":"bucket.instance","name":"gpxthc-31:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.40","timestamp":"2020-06-04T08:38:46.117432535Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.121915_884.1","section":"bucket.instance","name":"gpxthc-31:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.40","timestamp":"2020-06-04T08:38:46.121915138Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.124124_885.1","section":"bucket","name":"gpxthc-31","timestamp":"2020-06-04T08:38:46.124124794Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6ku9WQrR5WvQ3xnFGpcCP0M","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.129078_886.1","section":"bucket","name":"gpxthc-31","timestamp":"2020-06-04T08:38:46.129078803Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6ku9WQrR5WvQ3xnFGpcCP0M","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.175739_887.1","section":"bucket.instance","name":"gpxthc-32:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.41","timestamp":"2020-06-04T08:38:46.175739838Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.182982_888.1","section":"bucket.instance","name":"gpxthc-32:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.41","timestamp":"2020-06-04T08:38:46.182982979Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.186641_889.1","section":"bucket","name":"gpxthc-32","timestamp":"2020-06-04T08:38:46.186641231Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RUtCJbRmBTaCDOrE5uNNCNT","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.193307_890.1","section":"bucket","name":"gpxthc-32","timestamp":"2020-06-04T08:38:46.193307183Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RUtCJbRmBTaCDOrE5uNNCNT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.371575_892.1","section":"bucket.instance","name":"gpxthc-30:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.39","timestamp":"2020-06-04T08:38:46.371575852Z","data":{"read_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":1},"write_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":2},"status":{"status":"write"}}},{"id":"1_1591259926.377365_893.1","section":"bucket.instance","name":"gpxthc-30:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.39","timestamp":"2020-06-04T08:38:46.377365569Z","data":{"read_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":1},"write_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259926.431027_896.1","section":"bucket.instance","name":"gpxthc-31:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.40","timestamp":"2020-06-04T08:38:46.431027842Z","data":{"read_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":1},"write_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":2},"status":{"status":"write"}}},{"id":"1_1591259926.442905_897.1","section":"bucket.instance","name":"gpxthc-31:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.40","timestamp":"2020-06-04T08:38:46.442905741Z","data":{"read_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":1},"write_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259926.503230_898.1","section":"bucket.instance","name":"gpxthc-32:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.41","timestamp":"2020-06-04T08:38:46.503230913Z","data":{"read_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":1},"write_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":2},"status":{"status":"write"}}},{"id":"1_1591259926.513386_899.1","section":"bucket.instance","name":"gpxthc-32:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.41","timestamp":"2020-06-04T08:38:46.513386815Z","data":{"read_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":1},"write_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259926.597611_900.1","section":"bucket.instance","name":"gpxthc-33:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.42","timestamp":"2020-06-04T08:38:46.597611762Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.604278_901.1","section":"bucket.instance","name":"gpxthc-33:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.42","timestamp":"2020-06-04T08:38:46.604278594Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.611816_902.1","section":"bucket","name":"gpxthc-33","timestamp":"2020-06-04T08:38:46.611816568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TCD6G-8d3BbB8YgWl8O3Nwk","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.616028_903.1","section":"bucket","name":"gpxthc-33","timestamp":"2020-06-04T08:38:46.616028597Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TCD6G-8d3BbB8YgWl8O3Nwk","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.637171_904.1","section":"bucket.instance","name":"gpxthc-34:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.43","timestamp":"2020-06-04T08:38:46.637171077Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.641863_905.1","section":"bucket.instance","name":"gpxthc-34:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.43","timestamp":"2020-06-04T08:38:46.641863656Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.644434_906.1","section":"bucket","name":"gpxthc-34","timestamp":"2020-06-04T08:38:46.644434253Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P-m8zGy2fOnlsKo8No6u1Cq","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.652520_907.1","section":"bucket","name":"gpxthc-34","timestamp":"2020-06-04T08:38:46.652520435Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P-m8zGy2fOnlsKo8No6u1Cq","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.695989_908.1","section":"bucket.instance","name":"gpxthc-35:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.44","timestamp":"2020-06-04T08:38:46.695989087Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.700896_909.1","section":"bucket.instance","name":"gpxthc-35:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.44","timestamp":"2020-06-04T08:38:46.700896824Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.703158_910.1","section":"bucket","name":"gpxthc-35","timestamp":"2020-06-04T08:38:46.703158174Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gEvrMnPJFZrfyPUGCekqix_","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.707672_911.1","section":"bucket","name":"gpxthc-35","timestamp":"2020-06-04T08:38:46.707672554Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gEvrMnPJFZrfyPUGCekqix_","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.732054_912.1","section":"bucket.instance","name":"gpxthc-33:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.42","timestamp":"2020-06-04T08:38:46.732054711Z","data":{"read_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":1},"write_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":2},"status":{"status":"write"}}},{"id":"1_1591259926.737279_913.1","section":"bucket.instance","name":"gpxthc-33:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.42","timestamp":"2020-06-04T08:38:46.737279135Z","data":{"read_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":1},"write_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259927.790025_914.1","section":"bucket.instance","name":"gpxthc-34:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.43","timestamp":"2020-06-04T08:38:47.790025773Z","data":{"read_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":1},"write_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":2},"status":{"status":"write"}}},{"id":"1_1591259927.802447_915.1","section":"bucket.instance","name":"gpxthc-34:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.43","timestamp":"2020-06-04T08:38:47.802447374Z","data":{"read_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":1},"write_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259928.861627_916.1","section":"bucket.instance","name":"gpxthc-35:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.44","timestamp":"2020-06-04T08:38:48.861627553Z","data":{"read_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":1},"write_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":2},"status":{"status":"write"}}},{"id":"1_1591259928.873592_917.1","section":"bucket.instance","name":"gpxthc-35:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.44","timestamp":"2020-06-04T08:38:48.873592348Z","data":{"read_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":1},"write_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259928.954520_918.1","section":"bucket.instance","name":"gpxthc-36:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.45","timestamp":"2020-06-04T08:38:48.954520125Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_11-lNmUdWJkycuWy12Rkais","ver":1},"status":{"status":"write"}}},{"id":"1_1591259928.958725_919.1","section":"bucket.instance","name":"gpxthc-36:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.45","timestamp":"2020-06-04T08:38:48.958725871Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_11-lNmUdWJkycuWy12Rkais","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259928.960607_920.1","section":"bucket","name":"gpxthc-36","timestamp":"2020-06-04T08:38:48.960607582Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2lIEwet-R7Pmma_XC0XfJfn","ver":1},"status":{"status":"write"}}},{"id":"1_1591259928.964925_921.1","section":"bucket","name":"gpxthc-36","timestamp":"2020-06-04T08:38:48.964925190Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2lIEwet-R7Pmma_XC0XfJfn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259929.002482_922.1","section":"bucket.instance","name":"gpxthc-37:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.46","timestamp":"2020-06-04T08:38:49.002482783Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V-CEvpx3s4VyGthwNmj1yAU","ver":1},"status":{"status":"write"}}},{"id":"1_1591259929.007285_923.1","section":"bucket.instance","name":"gpxthc-37:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.46","timestamp":"2020-06-04T08:38:49.007285334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V-CEvpx3s4VyGthwNmj1yAU","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259929.009398_924.1","section":"bucket","name":"gpxthc-37","timestamp":"2020-06-04T08:38:49.009398457Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ek_-Wa_Uyu-XnjfASqAxdXy","ver":1},"status":{"status":"write"}}},{"id":"1_1591259929.013412_925.1","section":"bucket","name":"gpxthc-37","timestamp":"2020-06-04T08:38:49.013412886Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ek_-Wa_Uyu-XnjfASqAxdXy","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259929.052573_926.1","section":"bucket.instance","name":"gpxthc-38:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.47","timestamp":"2020-06-04T08:38:49.052573944Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N6EHuJJFZ7_2dQ30ZsrYxhQ","ver":1},"status":{"status":"write"}}},{"id":"1_1591259929.056846_927.1","section":"bucket.instance","name":"gpxthc-38:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.47","timestamp":"2020-06-04T08:38:49.056846763Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N6EHuJJFZ7_2dQ30ZsrYxhQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259929.059064_928.1","section":"bucket","name":"gpxthc-38","timestamp":"2020-06-04T08:38:49.059064075Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-zv4Cksv-zkyzdIqRjsaDb2","ver":1},"status":{"status":"write"}}},{"id":"1_1591259929.063266_929.1","section":"bucket","name":"gpxthc-38","timestamp":"2020-06-04T08:38:49.063266714Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-zv4Cksv-zkyzdIqRjsaDb2","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.609274_930.1","section":"bucket.instance","name":"gpxthc-39:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.48","timestamp":"2020-06-04T08:38:50.609274737Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VhICsf9zqk7f9jpddSNIqkP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.613351_932.1","section":"bucket.instance","name":"gpxthc-39:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.48","timestamp":"2020-06-04T08:38:50.613351600Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VhICsf9zqk7f9jpddSNIqkP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.615477_933.1","section":"bucket","name":"gpxthc-39","timestamp":"2020-06-04T08:38:50.615477264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v902jpCGpnCkP2gKKzOXdqt","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.619346_934.1","section":"bucket","name":"gpxthc-39","timestamp":"2020-06-04T08:38:50.619346741Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v902jpCGpnCkP2gKKzOXdqt","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.653610_935.1","section":"bucket.instance","name":"gpxthc-40:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.49","timestamp":"2020-06-04T08:38:50.653610015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pbAYVMP2sAf_BuaE5BYBTkr","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.658862_936.1","section":"bucket.instance","name":"gpxthc-40:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.49","timestamp":"2020-06-04T08:38:50.658862411Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pbAYVMP2sAf_BuaE5BYBTkr","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.661191_937.1","section":"bucket","name":"gpxthc-40","timestamp":"2020-06-04T08:38:50.661191135Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4cw4XeidBA2TPzMfHBQhnsn","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.665356_938.1","section":"bucket","name":"gpxthc-40","timestamp":"2020-06-04T08:38:50.665356122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4cw4XeidBA2TPzMfHBQhnsn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.711577_939.1","section":"bucket.instance","name":"gpxthc-41:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.50","timestamp":"2020-06-04T08:38:50.711577903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zFKqF0DR8qyD3-Tr3FbcjW1","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.719322_940.1","section":"bucket.instance","name":"gpxthc-41:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.50","timestamp":"2020-06-04T08:38:50.719322092Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zFKqF0DR8qyD3-Tr3FbcjW1","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.721242_941.1","section":"bucket","name":"gpxthc-41","timestamp":"2020-06-04T08:38:50.721242218Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n_mX6j5IFhQpmY51w3jLeA3","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.725192_942.1","section":"bucket","name":"gpxthc-41","timestamp":"2020-06-04T08:38:50.725192405Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n_mX6j5IFhQpmY51w3jLeA3","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259341.737046_34.1","section":"bucket.instance","name":"gpxthc-1:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.1","timestamp":"2020-06-04T08:29:01.737046909Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-VXJYi2_CLvrsqw6qc6DCr5","ver":1},"status":{"status":"write"}}},{"id":"1_1591259341.740440_35.1","section":"bucket.instance","name":"gpxthc-1:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.1","timestamp":"2020-06-04T08:29:01.740440081Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-VXJYi2_CLvrsqw6qc6DCr5","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259341.742105_36.1","section":"bucket","name":"gpxthc-1","timestamp":"2020-06-04T08:29:01.742105318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6GkvMPsGVWRJghpjcfPP5td","ver":1},"status":{"status":"write"}}},{"id":"1_1591259341.745009_37.1","section":"bucket","name":"gpxthc-1","timestamp":"2020-06-04T08:29:01.745009929Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6GkvMPsGVWRJghpjcfPP5td","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259341.768025_38.1","section":"bucket.instance","name":"gpxthc-2:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.2","timestamp":"2020-06-04T08:29:01.768025171Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XvNT082cvaVbqbr-2ZESWeT","ver":1},"status":{"status":"write"}}},{"id":"1_1591259341.771975_39.1","section":"bucket.instance","name":"gpxthc-2:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.2","timestamp":"2020-06-04T08:29:01.771975782Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XvNT082cvaVbqbr-2ZESWeT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259341.773834_40.1","section":"bucket","name":"gpxthc-2","timestamp":"2020-06-04T08:29:01.773834414Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AQGxlKt5b4yEuqgCWbWM_je","ver":1},"status":{"status":"write"}}},{"id":"1_1591259341.777323_41.1","section":"bucket","name":"gpxthc-2","timestamp":"2020-06-04T08:29:01.777323422Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AQGxlKt5b4yEuqgCWbWM_je","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259345.200120_42.1","section":"bucket.instance","name":"gpxthc-3:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.3","timestamp":"2020-06-04T08:29:05.200120296Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r--8CKmcPrYbCzDhO-F06sP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259345.203880_43.1","section":"bucket.instance","name":"gpxthc-3:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.3","timestamp":"2020-06-04T08:29:05.203880111Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r--8CKmcPrYbCzDhO-F06sP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259345.205531_44.1","section":"bucket","name":"gpxthc-3","timestamp":"2020-06-04T08:29:05.205531158Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bgP8ilt4JXWshxEQEKDTOxL","ver":1},"status":{"status":"write"}}},{"id":"1_1591259345.209069_45.1","section":"bucket","name":"gpxthc-3","timestamp":"2020-06-04T08:29:05.209069471Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bgP8ilt4JXWshxEQEKDTOxL","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.383145_46.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.4","timestamp":"2020-06-04T08:29:06.383145515Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HbOqQVlA66qYh_U-6gTxraM","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.388144_47.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.4","timestamp":"2020-06-04T08:29:06.388144053Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HbOqQVlA66qYh_U-6gTxraM","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.390123_48.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:06.390123332Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ThXePFlF36ZdSx-rFItfhQ8","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.394528_49.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:06.394528126Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ThXePFlF36ZdSx-rFItfhQ8","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.415142_50.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.5","timestamp":"2020-06-04T08:29:06.415142498Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FgaMM_80CprME0J7uTE6atP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.419201_51.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.5","timestamp":"2020-06-04T08:29:06.419201114Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FgaMM_80CprME0J7uTE6atP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.421086_52.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:06.421086802Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Nj0SyzQoWhfLBcf7quWXETq","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.424778_53.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:06.424778499Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Nj0SyzQoWhfLBcf7quWXETq","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.463965_54.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.6","timestamp":"2020-06-04T08:29:06.463965385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-cllnOXQdxzDqATQ1dbWkmx","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.467779_55.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.6","timestamp":"2020-06-04T08:29:06.467779786Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-cllnOXQdxzDqATQ1dbWkmx","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.469502_56.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:06.469502201Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uaSCzOL2DVEOK-odgsLCwFs","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.473034_57.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:06.473034822Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uaSCzOL2DVEOK-odgsLCwFs","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.614173_58.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.7","timestamp":"2020-06-04T08:29:07.614173009Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QW7H8a3CUThQ_bRH4yAqRKv","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.618181_59.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.7","timestamp":"2020-06-04T08:29:07.618181728Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QW7H8a3CUThQ_bRH4yAqRKv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.619881_60.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:07.619881218Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5sgvOjv3iB27roqBQ7cUZbh","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.623348_61.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:07.623348297Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5sgvOjv3iB27roqBQ7cUZbh","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259347.647460_62.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.8","timestamp":"2020-06-04T08:29:07.647460278Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eGGXzzZG1A9GEn7uWckVfr-","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.651908_63.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.8","timestamp":"2020-06-04T08:29:07.651908533Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eGGXzzZG1A9GEn7uWckVfr-","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.654187_64.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:07.654187090Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UJzl8Whz1V5QSBu3ohXNupT","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.657537_65.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:07.657537293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UJzl8Whz1V5QSBu3ohXNupT","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259347.686764_66.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.9","timestamp":"2020-06-04T08:29:07.686764572Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vZquMGd6hFx42B6ADczwHim","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.691403_67.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.9","timestamp":"2020-06-04T08:29:07.691403042Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vZquMGd6hFx42B6ADczwHim","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.693513_68.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:07.693513613Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UOipH4LH_UUuQBoXOTlPqdc","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.698070_69.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:07.698070773Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UOipH4LH_UUuQBoXOTlPqdc","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259347.733629_70.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.10","timestamp":"2020-06-04T08:29:07.733629268Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4NNS9aPc6N_RBBTATY5XJN5","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.737752_71.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.10","timestamp":"2020-06-04T08:29:07.737752160Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4NNS9aPc6N_RBBTATY5XJN5","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.739454_72.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:07.739454023Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9rrBHViPIaHB1oKA4-fy2a7","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.742970_73.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:07.742970942Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9rrBHViPIaHB1oKA4-fy2a7","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259347.792004_74.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.11","timestamp":"2020-06-04T08:29:07.792004579Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R8wjaedoiarBcdJxSdsNG9i","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.797167_75.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.11","timestamp":"2020-06-04T08:29:07.797167708Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R8wjaedoiarBcdJxSdsNG9i","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.798799_76.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:07.798799372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFqu-tIqXwMndR9GZsMEdUg","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.801827_77.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:07.801827701Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFqu-tIqXwMndR9GZsMEdUg","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259347.895642_78.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.12","timestamp":"2020-06-04T08:29:07.895642957Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_813CHCsdbVJhG-8Friwea0Q","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.900475_79.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.12","timestamp":"2020-06-04T08:29:07.900475349Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_813CHCsdbVJhG-8Friwea0Q","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.903751_80.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:07.903751639Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4jTRpPwqpn_U-wRoBFCv-5t","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.907394_81.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:07.907394318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4jTRpPwqpn_U-wRoBFCv-5t","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259348.003913_82.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.13","timestamp":"2020-06-04T08:29:08.003913702Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vjl-KiOIIS-mHj4Knkvyvbn","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.008410_83.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.13","timestamp":"2020-06-04T08:29:08.008410230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vjl-KiOIIS-mHj4Knkvyvbn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259348.010405_84.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:08.010405065Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YXdPZK993bwZgNRYKqmPKbg","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.013902_85.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:08.013902468Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YXdPZK993bwZgNRYKqmPKbg","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259348.061824_86.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.14","timestamp":"2020-06-04T08:29:08.061824583Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GmPkZfz9onYDYux9NvroVpb","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.066453_87.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.14","timestamp":"2020-06-04T08:29:08.066453149Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GmPkZfz9onYDYux9NvroVpb","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259348.068278_88.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:08.068278859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n5UFL0MW6wtQDqqBX5p9xOv","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.071562_89.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:08.071562898Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n5UFL0MW6wtQDqqBX5p9xOv","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259348.164678_90.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.15","timestamp":"2020-06-04T08:29:08.164678779Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wnNQvYf5A6zQTv_L1L16Vj_","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.169209_91.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.15","timestamp":"2020-06-04T08:29:08.169209509Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wnNQvYf5A6zQTv_L1L16Vj_","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259348.171482_92.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:08.171482220Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HuczynHZJw9zrk53biUsBon","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.175359_93.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:08.175359125Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HuczynHZJw9zrk53biUsBon","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259349.444497_94.1","section":"bucket.instance","name":"gpxthc-7:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.16","timestamp":"2020-06-04T08:29:09.444497949Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0hH3KdNQ7Ror_-PbjMomrOK","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.449423_95.1","section":"bucket.instance","name":"gpxthc-7:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.16","timestamp":"2020-06-04T08:29:09.449423359Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0hH3KdNQ7Ror_-PbjMomrOK","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259349.451676_96.1","section":"bucket","name":"gpxthc-7","timestamp":"2020-06-04T08:29:09.451676822Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.455775_97.1","section":"bucket","name":"gpxthc-7","timestamp":"2020-06-04T08:29:09.455775359Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259349.478408_98.1","section":"bucket.instance","name":"gpxthc-8:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.17","timestamp":"2020-06-04T08:29:09.478408677Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3tdK5SOwUxP7sixtd5Iiusu","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.483441_99.1","section":"bucket.instance","name":"gpxthc-8:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.17","timestamp":"2020-06-04T08:29:09.483441043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3tdK5SOwUxP7sixtd5Iiusu","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259349.485210_100.1","section":"bucket","name":"gpxthc-8","timestamp":"2020-06-04T08:29:09.485210362Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.489008_101.1","section":"bucket","name":"gpxthc-8","timestamp":"2020-06-04T08:29:09.489008069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259349.542500_102.1","section":"bucket.instance","name":"gpxthc-9:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.18","timestamp":"2020-06-04T08:29:09.542500556Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZM4c0G2TMLKguEZQHJhJ_U1","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.547391_103.1","section":"bucket.instance","name":"gpxthc-9:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.18","timestamp":"2020-06-04T08:29:09.547391664Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZM4c0G2TMLKguEZQHJhJ_U1","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259349.549662_104.1","section":"bucket","name":"gpxthc-9","timestamp":"2020-06-04T08:29:09.549662472Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.557874_105.1","section":"bucket","name":"gpxthc-9","timestamp":"2020-06-04T08:29:09.557874857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259350.809493_106.1","section":"bucket","name":"gpxthc-7","timestamp":"2020-06-04T08:29:10.809493171Z","data":{"read_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":1},"write_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":2},"status":{"status":"remove"}}},{"id":"1_1591259350.815979_107.1","section":"bucket","name":"gpxthc-7","timestamp":"2020-06-04T08:29:10.815979805Z","data":{"read_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":1},"write_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259350.882584_108.1","section":"bucket","name":"gpxthc-8","timestamp":"2020-06-04T08:29:10.882584350Z","data":{"read_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":1},"write_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":2},"status":{"status":"remove"}}},{"id":"1_1591259350.890059_109.1","section":"bucket","name":"gpxthc-8","timestamp":"2020-06-04T08:29:10.890059979Z","data":{"read_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":1},"write_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259350.978944_110.1","section":"bucket","name":"gpxthc-9","timestamp":"2020-06-04T08:29:10.978944722Z","data":{"read_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":1},"write_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":2},"status":{"status":"remove"}}},{"id":"1_1591259350.985454_111.1","section":"bucket","name":"gpxthc-9","timestamp":"2020-06-04T08:29:10.985454315Z","data":{"read_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":1},"write_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":2},"status":{"status":"complete"}}}] 2020-06-04T08:41:13.783 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '838ed6b2-fa2f-4917-a5c1-6c82bb405cda', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:13.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '838ed6b2-fa2f-4917-a5c1-6c82bb405cda', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:13.784 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 838ed6b2-fa2f-4917-a5c1-6c82bb405cda --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:13.824 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:13.825 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:14.073 INFO:teuthology.orchestra.run.smithi196.stdout:[{"id":"1_1591259219.822557_39.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-06-04T08:26:59.822557133Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UKgIH850zXzbqniMBbCDHLr","ver":1},"status":{"status":"write"}}},{"id":"1_1591259219.830770_41.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2020-06-04T08:26:59.830770292Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UKgIH850zXzbqniMBbCDHLr","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.183102_42.1","section":"bucket.instance","name":"gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19","timestamp":"2020-06-04T08:29:12.183102477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VfH48PXmsZggFiAsqBANxvt","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.187271_50.1","section":"bucket.instance","name":"gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19","timestamp":"2020-06-04T08:29:12.187271885Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VfH48PXmsZggFiAsqBANxvt","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.189031_51.1","section":"bucket","name":"gpxthc-10","timestamp":"2020-06-04T08:29:12.189031257Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_h8WUqJo-QErjBMZIQdgNCtn","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.192886_52.1","section":"bucket","name":"gpxthc-10","timestamp":"2020-06-04T08:29:12.192886901Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_h8WUqJo-QErjBMZIQdgNCtn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.214300_53.1","section":"bucket.instance","name":"gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20","timestamp":"2020-06-04T08:29:12.214300618Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OITM9jIqMd3jh6APsdWU31A","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.219348_54.1","section":"bucket.instance","name":"gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20","timestamp":"2020-06-04T08:29:12.219348485Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OITM9jIqMd3jh6APsdWU31A","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.221278_55.1","section":"bucket","name":"gpxthc-11","timestamp":"2020-06-04T08:29:12.221278412Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-nUpFbAACCD5rYXC7DcxUQR","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.225194_56.1","section":"bucket","name":"gpxthc-11","timestamp":"2020-06-04T08:29:12.225194355Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-nUpFbAACCD5rYXC7DcxUQR","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.268612_57.1","section":"bucket.instance","name":"gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21","timestamp":"2020-06-04T08:29:12.268612619Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zd8ii08n-ZihNb68aNArIKM","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.273166_58.1","section":"bucket.instance","name":"gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21","timestamp":"2020-06-04T08:29:12.273166922Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zd8ii08n-ZihNb68aNArIKM","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259352.275116_59.1","section":"bucket","name":"gpxthc-12","timestamp":"2020-06-04T08:29:12.275116572Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AleC4XiO8GV6Mta70FMehuN","ver":1},"status":{"status":"write"}}},{"id":"1_1591259352.279189_60.1","section":"bucket","name":"gpxthc-12","timestamp":"2020-06-04T08:29:12.279189055Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AleC4XiO8GV6Mta70FMehuN","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.619667_61.1","section":"bucket.instance","name":"gpxthc-13:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.22","timestamp":"2020-06-04T08:29:31.619667228Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G4zU5SP7vzOavY4UDRy0kVT","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.624339_68.1","section":"bucket.instance","name":"gpxthc-13:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.22","timestamp":"2020-06-04T08:29:31.624339358Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_G4zU5SP7vzOavY4UDRy0kVT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.626643_69.1","section":"bucket","name":"gpxthc-13","timestamp":"2020-06-04T08:29:31.626643743Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VUOAgr-b5NnlCfY5DmWhaqq","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.630935_70.1","section":"bucket","name":"gpxthc-13","timestamp":"2020-06-04T08:29:31.630935737Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VUOAgr-b5NnlCfY5DmWhaqq","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.653316_71.1","section":"bucket.instance","name":"gpxthc-14:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.23","timestamp":"2020-06-04T08:29:31.653316611Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YhqpqT7kUaddbo1-1KfMhvv","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.658400_72.1","section":"bucket.instance","name":"gpxthc-14:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.23","timestamp":"2020-06-04T08:29:31.658400301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YhqpqT7kUaddbo1-1KfMhvv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.660536_73.1","section":"bucket","name":"gpxthc-14","timestamp":"2020-06-04T08:29:31.660536384Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Br3NXT9kNBdoMbsZqQdcMrv","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.664400_74.1","section":"bucket","name":"gpxthc-14","timestamp":"2020-06-04T08:29:31.664400106Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Br3NXT9kNBdoMbsZqQdcMrv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.712373_75.1","section":"bucket.instance","name":"gpxthc-15:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.24","timestamp":"2020-06-04T08:29:31.712373958Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_frk3BKkQ_Qa4KdotA_oTnm0","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.716427_76.1","section":"bucket.instance","name":"gpxthc-15:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.24","timestamp":"2020-06-04T08:29:31.716427746Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_frk3BKkQ_Qa4KdotA_oTnm0","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259371.719782_77.1","section":"bucket","name":"gpxthc-15","timestamp":"2020-06-04T08:29:31.719782440Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CRGlFFvUIxHRKMSeAVvS4kF","ver":1},"status":{"status":"write"}}},{"id":"1_1591259371.724712_78.1","section":"bucket","name":"gpxthc-15","timestamp":"2020-06-04T08:29:31.724712205Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CRGlFFvUIxHRKMSeAVvS4kF","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.428231_79.1","section":"bucket.instance","name":"gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25","timestamp":"2020-06-04T08:30:16.428231743Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.433809_94.1","section":"bucket.instance","name":"gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25","timestamp":"2020-06-04T08:30:16.433809103Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.435876_95.1","section":"bucket","name":"gpxthc-16","timestamp":"2020-06-04T08:30:16.435876487Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4nNLQIaMSuVxPP12bYBinmN","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.440082_96.1","section":"bucket","name":"gpxthc-16","timestamp":"2020-06-04T08:30:16.440082082Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4nNLQIaMSuVxPP12bYBinmN","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.465814_97.1","section":"bucket.instance","name":"gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26","timestamp":"2020-06-04T08:30:16.465814342Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.470277_98.1","section":"bucket.instance","name":"gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26","timestamp":"2020-06-04T08:30:16.470277700Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.472163_99.1","section":"bucket","name":"gpxthc-17","timestamp":"2020-06-04T08:30:16.472163913Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jIqc-jlMoWLqci8_G-SuMb2","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.476487_100.1","section":"bucket","name":"gpxthc-17","timestamp":"2020-06-04T08:30:16.476487448Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jIqc-jlMoWLqci8_G-SuMb2","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.517171_101.1","section":"bucket.instance","name":"gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27","timestamp":"2020-06-04T08:30:16.517171806Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.522567_102.1","section":"bucket.instance","name":"gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27","timestamp":"2020-06-04T08:30:16.522567636Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.524592_103.1","section":"bucket","name":"gpxthc-18","timestamp":"2020-06-04T08:30:16.524592149Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T8PKYfLd2MscBs2JJLeNmbu","ver":1},"status":{"status":"write"}}},{"id":"1_1591259416.529893_104.1","section":"bucket","name":"gpxthc-18","timestamp":"2020-06-04T08:30:16.529893092Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T8PKYfLd2MscBs2JJLeNmbu","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259416.564959_105.1","section":"bucket.instance","name":"gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25","timestamp":"2020-06-04T08:30:16.564959314Z","data":{"read_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":1},"write_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":2},"status":{"status":"write"}}},{"id":"1_1591259416.569413_106.1","section":"bucket.instance","name":"gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25","timestamp":"2020-06-04T08:30:16.569413363Z","data":{"read_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":1},"write_version":{"tag":"_nxJH1DUi1FGr8obcc4qHjYM","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259416.573339_107.1","section":"bucket.instance","name":"gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26","timestamp":"2020-06-04T08:30:16.573339571Z","data":{"read_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":1},"write_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":2},"status":{"status":"write"}}},{"id":"1_1591259416.578468_108.1","section":"bucket.instance","name":"gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26","timestamp":"2020-06-04T08:30:16.578468486Z","data":{"read_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":1},"write_version":{"tag":"_z5yxYFmKS3u6HXO8_WfDwA3","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259416.593653_109.1","section":"bucket.instance","name":"gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27","timestamp":"2020-06-04T08:30:16.593653914Z","data":{"read_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":1},"write_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":2},"status":{"status":"write"}}},{"id":"1_1591259416.599911_110.1","section":"bucket.instance","name":"gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27","timestamp":"2020-06-04T08:30:16.599911549Z","data":{"read_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":1},"write_version":{"tag":"_GQoV8jlFx4eVw2BoYdhK06V","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259628.323783_111.1","section":"bucket.instance","name":"gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28","timestamp":"2020-06-04T08:33:48.323783159Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":1},"status":{"status":"write"}}},{"id":"1_1591259628.328728_148.1","section":"bucket.instance","name":"gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28","timestamp":"2020-06-04T08:33:48.328728853Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259628.330710_149.1","section":"bucket","name":"gpxthc-19","timestamp":"2020-06-04T08:33:48.330710900Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_93QBHlBvEVUFT9b_wxOIMnq","ver":1},"status":{"status":"write"}}},{"id":"1_1591259628.334787_150.1","section":"bucket","name":"gpxthc-19","timestamp":"2020-06-04T08:33:48.334787040Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_93QBHlBvEVUFT9b_wxOIMnq","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259628.341176_152.1","section":"bucket.instance","name":"gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28","timestamp":"2020-06-04T08:33:48.341176479Z","data":{"read_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":1},"write_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":2},"status":{"status":"write"}}},{"id":"1_1591259628.347410_153.1","section":"bucket.instance","name":"gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28","timestamp":"2020-06-04T08:33:48.347410493Z","data":{"read_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":1},"write_version":{"tag":"_p4Bf-hXChpQC-BzmwL5T-C4","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259767.841895_739.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:36:07.841895512Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":1},"status":{"status":"write"}}},{"id":"1_1591259767.846376_745.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:36:07.846376164Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259767.848773_746.1","section":"bucket","name":"gpxthc-20","timestamp":"2020-06-04T08:36:07.848773543Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ErTLIwy11AZgBe_n6j9EOvj","ver":1},"status":{"status":"write"}}},{"id":"1_1591259767.852808_747.1","section":"bucket","name":"gpxthc-20","timestamp":"2020-06-04T08:36:07.852808654Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ErTLIwy11AZgBe_n6j9EOvj","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259774.352869_748.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:36:14.352869429Z","data":{"read_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":1},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":2},"status":{"status":"write"}}},{"id":"1_1591259774.363917_755.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:36:14.363917038Z","data":{"read_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":1},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259833.992686_770.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:37:13.992686192Z","data":{"read_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":2},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":3},"status":{"status":"write"}}},{"id":"1_1591259834.004633_772.1","section":"bucket.instance","name":"gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29","timestamp":"2020-06-04T08:37:14.004633298Z","data":{"read_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":2},"write_version":{"tag":"_E5jhInqVBgNl-VQOE5p0r2H","ver":3},"status":{"status":"complete"}}},{"id":"1_1591259877.840447_773.1","section":"bucket.instance","name":"gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30","timestamp":"2020-06-04T08:37:57.840447936Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.844931_787.1","section":"bucket.instance","name":"gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30","timestamp":"2020-06-04T08:37:57.844931870Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.847150_788.1","section":"bucket","name":"gpxthc-21","timestamp":"2020-06-04T08:37:57.847150642Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N959Mvlm6EfOalZxV2Ys2gP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.851202_789.1","section":"bucket","name":"gpxthc-21","timestamp":"2020-06-04T08:37:57.851202385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N959Mvlm6EfOalZxV2Ys2gP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.876704_790.1","section":"bucket.instance","name":"gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31","timestamp":"2020-06-04T08:37:57.876704072Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.881398_791.1","section":"bucket.instance","name":"gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31","timestamp":"2020-06-04T08:37:57.881398442Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.883373_792.1","section":"bucket","name":"gpxthc-22","timestamp":"2020-06-04T08:37:57.883373243Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dEQaV8gUPyD0QSgk_KOV4um","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.887399_793.1","section":"bucket","name":"gpxthc-22","timestamp":"2020-06-04T08:37:57.887399447Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dEQaV8gUPyD0QSgk_KOV4um","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.931319_794.1","section":"bucket.instance","name":"gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32","timestamp":"2020-06-04T08:37:57.931319262Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.936911_795.1","section":"bucket.instance","name":"gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32","timestamp":"2020-06-04T08:37:57.936911048Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.939469_796.1","section":"bucket","name":"gpxthc-23","timestamp":"2020-06-04T08:37:57.939469678Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c3fe5iwyxyrYzoUjllCpsvU","ver":1},"status":{"status":"write"}}},{"id":"1_1591259877.943713_797.1","section":"bucket","name":"gpxthc-23","timestamp":"2020-06-04T08:37:57.943713180Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_c3fe5iwyxyrYzoUjllCpsvU","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259877.969944_798.1","section":"bucket.instance","name":"gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30","timestamp":"2020-06-04T08:37:57.969944576Z","data":{"read_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":1},"write_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":2},"status":{"status":"write"}}},{"id":"1_1591259877.980272_799.1","section":"bucket.instance","name":"gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30","timestamp":"2020-06-04T08:37:57.980272398Z","data":{"read_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":1},"write_version":{"tag":"_Y8lCr7iI9eh-fNM0pZFU2P9","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259877.984533_800.1","section":"bucket.instance","name":"gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31","timestamp":"2020-06-04T08:37:57.984533518Z","data":{"read_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":1},"write_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":2},"status":{"status":"write"}}},{"id":"1_1591259877.989050_801.1","section":"bucket.instance","name":"gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31","timestamp":"2020-06-04T08:37:57.989050456Z","data":{"read_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":1},"write_version":{"tag":"_mS3sSOgVIXSYfffVewyiGDv","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259878.000419_802.1","section":"bucket.instance","name":"gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32","timestamp":"2020-06-04T08:37:58.000419669Z","data":{"read_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":1},"write_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":2},"status":{"status":"write"}}},{"id":"1_1591259878.006196_803.1","section":"bucket.instance","name":"gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32","timestamp":"2020-06-04T08:37:58.006196070Z","data":{"read_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":1},"write_version":{"tag":"_FoZn4tiF6hx6SH4OcKFRNCf","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259904.804937_804.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.804937045Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.810097_821.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.810097096Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.812274_822.1","section":"bucket","name":"gpxthc-24","timestamp":"2020-06-04T08:38:24.812274099Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RQ8eJ9g70-tjyJRj0QIilAr","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.816235_823.1","section":"bucket","name":"gpxthc-24","timestamp":"2020-06-04T08:38:24.816235964Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RQ8eJ9g70-tjyJRj0QIilAr","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.846016_824.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:24.846016920Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.850663_825.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:24.850663462Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.852772_826.1","section":"bucket","name":"gpxthc-25","timestamp":"2020-06-04T08:38:24.852772198Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nSe4ph9lkroFUUb_S85OnlC","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.857137_827.1","section":"bucket","name":"gpxthc-25","timestamp":"2020-06-04T08:38:24.857137822Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_nSe4ph9lkroFUUb_S85OnlC","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.906938_828.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:24.906938522Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.912058_829.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:24.912058812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.914274_830.1","section":"bucket","name":"gpxthc-26","timestamp":"2020-06-04T08:38:24.914274818Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_edlwuY9ktEE0aOm2248enxI","ver":1},"status":{"status":"write"}}},{"id":"1_1591259904.918216_831.1","section":"bucket","name":"gpxthc-26","timestamp":"2020-06-04T08:38:24.918216849Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_edlwuY9ktEE0aOm2248enxI","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259904.939673_832.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.939673512Z","data":{"read_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":1},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":2},"status":{"status":"write"}}},{"id":"1_1591259904.945900_833.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.945900417Z","data":{"read_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":1},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259904.952407_834.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.952407248Z","data":{"read_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":2},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":3},"status":{"status":"write"}}},{"id":"1_1591259904.959605_835.1","section":"bucket.instance","name":"gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33","timestamp":"2020-06-04T08:38:24.959605042Z","data":{"read_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":2},"write_version":{"tag":"_m2jGaedrMn3yhjVIsVmfwSp","ver":3},"status":{"status":"complete"}}},{"id":"1_1591259904.969679_836.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:24.969679222Z","data":{"read_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":1},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":2},"status":{"status":"write"}}},{"id":"1_1591259904.981104_837.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:24.981104198Z","data":{"read_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":1},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259905.015511_838.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:25.015511493Z","data":{"read_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":2},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":3},"status":{"status":"write"}}},{"id":"1_1591259905.023462_839.1","section":"bucket.instance","name":"gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34","timestamp":"2020-06-04T08:38:25.023462062Z","data":{"read_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":2},"write_version":{"tag":"_Dul9ntLgmtyYYheOcX5mAG6","ver":3},"status":{"status":"complete"}}},{"id":"1_1591259905.044506_840.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:25.044506828Z","data":{"read_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":1},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":2},"status":{"status":"write"}}},{"id":"1_1591259905.049929_841.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:25.049929340Z","data":{"read_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":1},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259905.058537_842.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:25.058537498Z","data":{"read_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":2},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":3},"status":{"status":"write"}}},{"id":"1_1591259905.064571_843.1","section":"bucket.instance","name":"gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35","timestamp":"2020-06-04T08:38:25.064571488Z","data":{"read_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":2},"write_version":{"tag":"__oye-kwN4gXLEjSBQH8NwBD","ver":3},"status":{"status":"complete"}}},{"id":"1_1591259925.725606_844.1","section":"bucket.instance","name":"gpxthc-27:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.36","timestamp":"2020-06-04T08:38:45.725606301Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.729923_861.1","section":"bucket.instance","name":"gpxthc-27:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.36","timestamp":"2020-06-04T08:38:45.729923565Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.732041_862.1","section":"bucket","name":"gpxthc-27","timestamp":"2020-06-04T08:38:45.732041328Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pBr-gACv7tk1mBrJwyLeMiI","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.736204_863.1","section":"bucket","name":"gpxthc-27","timestamp":"2020-06-04T08:38:45.736204921Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pBr-gACv7tk1mBrJwyLeMiI","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.764022_864.1","section":"bucket.instance","name":"gpxthc-28:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.37","timestamp":"2020-06-04T08:38:45.764022725Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.768419_865.1","section":"bucket.instance","name":"gpxthc-28:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.37","timestamp":"2020-06-04T08:38:45.768419481Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.770593_866.1","section":"bucket","name":"gpxthc-28","timestamp":"2020-06-04T08:38:45.770593129Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YbhL-HGVrfTpz3aCSgkZ-1i","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.775197_867.1","section":"bucket","name":"gpxthc-28","timestamp":"2020-06-04T08:38:45.775197377Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YbhL-HGVrfTpz3aCSgkZ-1i","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.820803_868.1","section":"bucket.instance","name":"gpxthc-29:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.38","timestamp":"2020-06-04T08:38:45.820803858Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.825390_869.1","section":"bucket.instance","name":"gpxthc-29:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.38","timestamp":"2020-06-04T08:38:45.825390920Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.827444_870.1","section":"bucket","name":"gpxthc-29","timestamp":"2020-06-04T08:38:45.827444882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HJvkKg7YYuNuo_RQ2C1dEVG","ver":1},"status":{"status":"write"}}},{"id":"1_1591259925.831901_871.1","section":"bucket","name":"gpxthc-29","timestamp":"2020-06-04T08:38:45.831901352Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HJvkKg7YYuNuo_RQ2C1dEVG","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259925.856470_872.1","section":"bucket.instance","name":"gpxthc-27:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.36","timestamp":"2020-06-04T08:38:45.856470724Z","data":{"read_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":1},"write_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":2},"status":{"status":"write"}}},{"id":"1_1591259925.861409_873.1","section":"bucket.instance","name":"gpxthc-27:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.36","timestamp":"2020-06-04T08:38:45.861409642Z","data":{"read_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":1},"write_version":{"tag":"_ANztt3mCOpQX-7J4zvG1hEP","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259925.913880_874.1","section":"bucket.instance","name":"gpxthc-28:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.37","timestamp":"2020-06-04T08:38:45.913880548Z","data":{"read_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":1},"write_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":2},"status":{"status":"write"}}},{"id":"1_1591259925.924621_875.1","section":"bucket.instance","name":"gpxthc-28:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.37","timestamp":"2020-06-04T08:38:45.924621260Z","data":{"read_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":1},"write_version":{"tag":"_P_s4aPJ0bgMHYQ9-MGfhmj4","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259925.983236_876.1","section":"bucket.instance","name":"gpxthc-29:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.38","timestamp":"2020-06-04T08:38:45.983236325Z","data":{"read_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":1},"write_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":2},"status":{"status":"write"}}},{"id":"1_1591259925.996695_877.1","section":"bucket.instance","name":"gpxthc-29:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.38","timestamp":"2020-06-04T08:38:45.996695629Z","data":{"read_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":1},"write_version":{"tag":"_H11jFhijkccvFL7HSvuuV1n","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259926.077449_878.1","section":"bucket.instance","name":"gpxthc-30:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.39","timestamp":"2020-06-04T08:38:46.077449382Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.083000_879.1","section":"bucket.instance","name":"gpxthc-30:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.39","timestamp":"2020-06-04T08:38:46.083000564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.085045_880.1","section":"bucket","name":"gpxthc-30","timestamp":"2020-06-04T08:38:46.085045274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jthV6t1d3f5V-3IMp3Qs5U2","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.088922_881.1","section":"bucket","name":"gpxthc-30","timestamp":"2020-06-04T08:38:46.088922963Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jthV6t1d3f5V-3IMp3Qs5U2","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.117432_883.1","section":"bucket.instance","name":"gpxthc-31:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.40","timestamp":"2020-06-04T08:38:46.117432535Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.121915_884.1","section":"bucket.instance","name":"gpxthc-31:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.40","timestamp":"2020-06-04T08:38:46.121915138Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.124124_885.1","section":"bucket","name":"gpxthc-31","timestamp":"2020-06-04T08:38:46.124124794Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6ku9WQrR5WvQ3xnFGpcCP0M","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.129078_886.1","section":"bucket","name":"gpxthc-31","timestamp":"2020-06-04T08:38:46.129078803Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6ku9WQrR5WvQ3xnFGpcCP0M","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.175739_887.1","section":"bucket.instance","name":"gpxthc-32:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.41","timestamp":"2020-06-04T08:38:46.175739838Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.182982_888.1","section":"bucket.instance","name":"gpxthc-32:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.41","timestamp":"2020-06-04T08:38:46.182982979Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.186641_889.1","section":"bucket","name":"gpxthc-32","timestamp":"2020-06-04T08:38:46.186641231Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RUtCJbRmBTaCDOrE5uNNCNT","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.193307_890.1","section":"bucket","name":"gpxthc-32","timestamp":"2020-06-04T08:38:46.193307183Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_RUtCJbRmBTaCDOrE5uNNCNT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.371575_892.1","section":"bucket.instance","name":"gpxthc-30:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.39","timestamp":"2020-06-04T08:38:46.371575852Z","data":{"read_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":1},"write_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":2},"status":{"status":"write"}}},{"id":"1_1591259926.377365_893.1","section":"bucket.instance","name":"gpxthc-30:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.39","timestamp":"2020-06-04T08:38:46.377365569Z","data":{"read_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":1},"write_version":{"tag":"_EvYja6umwjtgaeIXDq4oJmH","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259926.431027_896.1","section":"bucket.instance","name":"gpxthc-31:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.40","timestamp":"2020-06-04T08:38:46.431027842Z","data":{"read_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":1},"write_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":2},"status":{"status":"write"}}},{"id":"1_1591259926.442905_897.1","section":"bucket.instance","name":"gpxthc-31:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.40","timestamp":"2020-06-04T08:38:46.442905741Z","data":{"read_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":1},"write_version":{"tag":"_i0RE0aBPkzM4steT2QaLrj7","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259926.503230_898.1","section":"bucket.instance","name":"gpxthc-32:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.41","timestamp":"2020-06-04T08:38:46.503230913Z","data":{"read_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":1},"write_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":2},"status":{"status":"write"}}},{"id":"1_1591259926.513386_899.1","section":"bucket.instance","name":"gpxthc-32:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.41","timestamp":"2020-06-04T08:38:46.513386815Z","data":{"read_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":1},"write_version":{"tag":"_vGsTB3ojV7cpnZAodm9PQTC","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259926.597611_900.1","section":"bucket.instance","name":"gpxthc-33:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.42","timestamp":"2020-06-04T08:38:46.597611762Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.604278_901.1","section":"bucket.instance","name":"gpxthc-33:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.42","timestamp":"2020-06-04T08:38:46.604278594Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.611816_902.1","section":"bucket","name":"gpxthc-33","timestamp":"2020-06-04T08:38:46.611816568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TCD6G-8d3BbB8YgWl8O3Nwk","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.616028_903.1","section":"bucket","name":"gpxthc-33","timestamp":"2020-06-04T08:38:46.616028597Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TCD6G-8d3BbB8YgWl8O3Nwk","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.637171_904.1","section":"bucket.instance","name":"gpxthc-34:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.43","timestamp":"2020-06-04T08:38:46.637171077Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.641863_905.1","section":"bucket.instance","name":"gpxthc-34:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.43","timestamp":"2020-06-04T08:38:46.641863656Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.644434_906.1","section":"bucket","name":"gpxthc-34","timestamp":"2020-06-04T08:38:46.644434253Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P-m8zGy2fOnlsKo8No6u1Cq","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.652520_907.1","section":"bucket","name":"gpxthc-34","timestamp":"2020-06-04T08:38:46.652520435Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_P-m8zGy2fOnlsKo8No6u1Cq","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.695989_908.1","section":"bucket.instance","name":"gpxthc-35:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.44","timestamp":"2020-06-04T08:38:46.695989087Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.700896_909.1","section":"bucket.instance","name":"gpxthc-35:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.44","timestamp":"2020-06-04T08:38:46.700896824Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.703158_910.1","section":"bucket","name":"gpxthc-35","timestamp":"2020-06-04T08:38:46.703158174Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gEvrMnPJFZrfyPUGCekqix_","ver":1},"status":{"status":"write"}}},{"id":"1_1591259926.707672_911.1","section":"bucket","name":"gpxthc-35","timestamp":"2020-06-04T08:38:46.707672554Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gEvrMnPJFZrfyPUGCekqix_","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259926.732054_912.1","section":"bucket.instance","name":"gpxthc-33:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.42","timestamp":"2020-06-04T08:38:46.732054711Z","data":{"read_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":1},"write_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":2},"status":{"status":"write"}}},{"id":"1_1591259926.737279_913.1","section":"bucket.instance","name":"gpxthc-33:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.42","timestamp":"2020-06-04T08:38:46.737279135Z","data":{"read_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":1},"write_version":{"tag":"_kyKwtj9lnWpBDkYcPrYW4-J","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259927.790025_914.1","section":"bucket.instance","name":"gpxthc-34:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.43","timestamp":"2020-06-04T08:38:47.790025773Z","data":{"read_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":1},"write_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":2},"status":{"status":"write"}}},{"id":"1_1591259927.802447_915.1","section":"bucket.instance","name":"gpxthc-34:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.43","timestamp":"2020-06-04T08:38:47.802447374Z","data":{"read_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":1},"write_version":{"tag":"_kcjTCPvdY026LqoLiQ-08mX","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259928.861627_916.1","section":"bucket.instance","name":"gpxthc-35:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.44","timestamp":"2020-06-04T08:38:48.861627553Z","data":{"read_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":1},"write_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":2},"status":{"status":"write"}}},{"id":"1_1591259928.873592_917.1","section":"bucket.instance","name":"gpxthc-35:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.44","timestamp":"2020-06-04T08:38:48.873592348Z","data":{"read_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":1},"write_version":{"tag":"_YYTwZZIKjFLjjv3y9LBvvkd","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259928.954520_918.1","section":"bucket.instance","name":"gpxthc-36:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.45","timestamp":"2020-06-04T08:38:48.954520125Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_11-lNmUdWJkycuWy12Rkais","ver":1},"status":{"status":"write"}}},{"id":"1_1591259928.958725_919.1","section":"bucket.instance","name":"gpxthc-36:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.45","timestamp":"2020-06-04T08:38:48.958725871Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_11-lNmUdWJkycuWy12Rkais","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259928.960607_920.1","section":"bucket","name":"gpxthc-36","timestamp":"2020-06-04T08:38:48.960607582Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2lIEwet-R7Pmma_XC0XfJfn","ver":1},"status":{"status":"write"}}},{"id":"1_1591259928.964925_921.1","section":"bucket","name":"gpxthc-36","timestamp":"2020-06-04T08:38:48.964925190Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2lIEwet-R7Pmma_XC0XfJfn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259929.002482_922.1","section":"bucket.instance","name":"gpxthc-37:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.46","timestamp":"2020-06-04T08:38:49.002482783Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V-CEvpx3s4VyGthwNmj1yAU","ver":1},"status":{"status":"write"}}},{"id":"1_1591259929.007285_923.1","section":"bucket.instance","name":"gpxthc-37:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.46","timestamp":"2020-06-04T08:38:49.007285334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V-CEvpx3s4VyGthwNmj1yAU","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259929.009398_924.1","section":"bucket","name":"gpxthc-37","timestamp":"2020-06-04T08:38:49.009398457Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ek_-Wa_Uyu-XnjfASqAxdXy","ver":1},"status":{"status":"write"}}},{"id":"1_1591259929.013412_925.1","section":"bucket","name":"gpxthc-37","timestamp":"2020-06-04T08:38:49.013412886Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Ek_-Wa_Uyu-XnjfASqAxdXy","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259929.052573_926.1","section":"bucket.instance","name":"gpxthc-38:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.47","timestamp":"2020-06-04T08:38:49.052573944Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N6EHuJJFZ7_2dQ30ZsrYxhQ","ver":1},"status":{"status":"write"}}},{"id":"1_1591259929.056846_927.1","section":"bucket.instance","name":"gpxthc-38:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.47","timestamp":"2020-06-04T08:38:49.056846763Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_N6EHuJJFZ7_2dQ30ZsrYxhQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259929.059064_928.1","section":"bucket","name":"gpxthc-38","timestamp":"2020-06-04T08:38:49.059064075Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-zv4Cksv-zkyzdIqRjsaDb2","ver":1},"status":{"status":"write"}}},{"id":"1_1591259929.063266_929.1","section":"bucket","name":"gpxthc-38","timestamp":"2020-06-04T08:38:49.063266714Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-zv4Cksv-zkyzdIqRjsaDb2","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.609274_930.1","section":"bucket.instance","name":"gpxthc-39:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.48","timestamp":"2020-06-04T08:38:50.609274737Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VhICsf9zqk7f9jpddSNIqkP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.613351_932.1","section":"bucket.instance","name":"gpxthc-39:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.48","timestamp":"2020-06-04T08:38:50.613351600Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VhICsf9zqk7f9jpddSNIqkP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.615477_933.1","section":"bucket","name":"gpxthc-39","timestamp":"2020-06-04T08:38:50.615477264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v902jpCGpnCkP2gKKzOXdqt","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.619346_934.1","section":"bucket","name":"gpxthc-39","timestamp":"2020-06-04T08:38:50.619346741Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v902jpCGpnCkP2gKKzOXdqt","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.653610_935.1","section":"bucket.instance","name":"gpxthc-40:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.49","timestamp":"2020-06-04T08:38:50.653610015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pbAYVMP2sAf_BuaE5BYBTkr","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.658862_936.1","section":"bucket.instance","name":"gpxthc-40:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.49","timestamp":"2020-06-04T08:38:50.658862411Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pbAYVMP2sAf_BuaE5BYBTkr","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.661191_937.1","section":"bucket","name":"gpxthc-40","timestamp":"2020-06-04T08:38:50.661191135Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4cw4XeidBA2TPzMfHBQhnsn","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.665356_938.1","section":"bucket","name":"gpxthc-40","timestamp":"2020-06-04T08:38:50.665356122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4cw4XeidBA2TPzMfHBQhnsn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.711577_939.1","section":"bucket.instance","name":"gpxthc-41:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.50","timestamp":"2020-06-04T08:38:50.711577903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zFKqF0DR8qyD3-Tr3FbcjW1","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.719322_940.1","section":"bucket.instance","name":"gpxthc-41:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.50","timestamp":"2020-06-04T08:38:50.719322092Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zFKqF0DR8qyD3-Tr3FbcjW1","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259930.721242_941.1","section":"bucket","name":"gpxthc-41","timestamp":"2020-06-04T08:38:50.721242218Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n_mX6j5IFhQpmY51w3jLeA3","ver":1},"status":{"status":"write"}}},{"id":"1_1591259930.725192_942.1","section":"bucket","name":"gpxthc-41","timestamp":"2020-06-04T08:38:50.725192405Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n_mX6j5IFhQpmY51w3jLeA3","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259341.737046_34.1","section":"bucket.instance","name":"gpxthc-1:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.1","timestamp":"2020-06-04T08:29:01.737046909Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-VXJYi2_CLvrsqw6qc6DCr5","ver":1},"status":{"status":"write"}}},{"id":"1_1591259341.740440_35.1","section":"bucket.instance","name":"gpxthc-1:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.1","timestamp":"2020-06-04T08:29:01.740440081Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-VXJYi2_CLvrsqw6qc6DCr5","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259341.742105_36.1","section":"bucket","name":"gpxthc-1","timestamp":"2020-06-04T08:29:01.742105318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6GkvMPsGVWRJghpjcfPP5td","ver":1},"status":{"status":"write"}}},{"id":"1_1591259341.745009_37.1","section":"bucket","name":"gpxthc-1","timestamp":"2020-06-04T08:29:01.745009929Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6GkvMPsGVWRJghpjcfPP5td","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259341.768025_38.1","section":"bucket.instance","name":"gpxthc-2:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.2","timestamp":"2020-06-04T08:29:01.768025171Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XvNT082cvaVbqbr-2ZESWeT","ver":1},"status":{"status":"write"}}},{"id":"1_1591259341.771975_39.1","section":"bucket.instance","name":"gpxthc-2:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.2","timestamp":"2020-06-04T08:29:01.771975782Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XvNT082cvaVbqbr-2ZESWeT","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259341.773834_40.1","section":"bucket","name":"gpxthc-2","timestamp":"2020-06-04T08:29:01.773834414Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AQGxlKt5b4yEuqgCWbWM_je","ver":1},"status":{"status":"write"}}},{"id":"1_1591259341.777323_41.1","section":"bucket","name":"gpxthc-2","timestamp":"2020-06-04T08:29:01.777323422Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AQGxlKt5b4yEuqgCWbWM_je","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259345.200120_42.1","section":"bucket.instance","name":"gpxthc-3:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.3","timestamp":"2020-06-04T08:29:05.200120296Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r--8CKmcPrYbCzDhO-F06sP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259345.203880_43.1","section":"bucket.instance","name":"gpxthc-3:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.3","timestamp":"2020-06-04T08:29:05.203880111Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r--8CKmcPrYbCzDhO-F06sP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259345.205531_44.1","section":"bucket","name":"gpxthc-3","timestamp":"2020-06-04T08:29:05.205531158Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bgP8ilt4JXWshxEQEKDTOxL","ver":1},"status":{"status":"write"}}},{"id":"1_1591259345.209069_45.1","section":"bucket","name":"gpxthc-3","timestamp":"2020-06-04T08:29:05.209069471Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bgP8ilt4JXWshxEQEKDTOxL","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.383145_46.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.4","timestamp":"2020-06-04T08:29:06.383145515Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HbOqQVlA66qYh_U-6gTxraM","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.388144_47.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.4","timestamp":"2020-06-04T08:29:06.388144053Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HbOqQVlA66qYh_U-6gTxraM","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.390123_48.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:06.390123332Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ThXePFlF36ZdSx-rFItfhQ8","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.394528_49.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:06.394528126Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ThXePFlF36ZdSx-rFItfhQ8","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.415142_50.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.5","timestamp":"2020-06-04T08:29:06.415142498Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FgaMM_80CprME0J7uTE6atP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.419201_51.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.5","timestamp":"2020-06-04T08:29:06.419201114Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_FgaMM_80CprME0J7uTE6atP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.421086_52.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:06.421086802Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Nj0SyzQoWhfLBcf7quWXETq","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.424778_53.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:06.424778499Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Nj0SyzQoWhfLBcf7quWXETq","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.463965_54.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.6","timestamp":"2020-06-04T08:29:06.463965385Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-cllnOXQdxzDqATQ1dbWkmx","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.467779_55.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.6","timestamp":"2020-06-04T08:29:06.467779786Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-cllnOXQdxzDqATQ1dbWkmx","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259346.469502_56.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:06.469502201Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uaSCzOL2DVEOK-odgsLCwFs","ver":1},"status":{"status":"write"}}},{"id":"1_1591259346.473034_57.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:06.473034822Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uaSCzOL2DVEOK-odgsLCwFs","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.614173_58.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.7","timestamp":"2020-06-04T08:29:07.614173009Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QW7H8a3CUThQ_bRH4yAqRKv","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.618181_59.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.7","timestamp":"2020-06-04T08:29:07.618181728Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QW7H8a3CUThQ_bRH4yAqRKv","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.619881_60.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:07.619881218Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5sgvOjv3iB27roqBQ7cUZbh","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.623348_61.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:07.623348297Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5sgvOjv3iB27roqBQ7cUZbh","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259347.647460_62.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.8","timestamp":"2020-06-04T08:29:07.647460278Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eGGXzzZG1A9GEn7uWckVfr-","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.651908_63.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.8","timestamp":"2020-06-04T08:29:07.651908533Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eGGXzzZG1A9GEn7uWckVfr-","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.654187_64.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:07.654187090Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UJzl8Whz1V5QSBu3ohXNupT","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.657537_65.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:07.657537293Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UJzl8Whz1V5QSBu3ohXNupT","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259347.686764_66.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.9","timestamp":"2020-06-04T08:29:07.686764572Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vZquMGd6hFx42B6ADczwHim","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.691403_67.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.9","timestamp":"2020-06-04T08:29:07.691403042Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vZquMGd6hFx42B6ADczwHim","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.693513_68.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:07.693513613Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UOipH4LH_UUuQBoXOTlPqdc","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.698070_69.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:07.698070773Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UOipH4LH_UUuQBoXOTlPqdc","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259347.733629_70.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.10","timestamp":"2020-06-04T08:29:07.733629268Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4NNS9aPc6N_RBBTATY5XJN5","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.737752_71.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.10","timestamp":"2020-06-04T08:29:07.737752160Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4NNS9aPc6N_RBBTATY5XJN5","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.739454_72.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:07.739454023Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9rrBHViPIaHB1oKA4-fy2a7","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.742970_73.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:07.742970942Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9rrBHViPIaHB1oKA4-fy2a7","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259347.792004_74.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.11","timestamp":"2020-06-04T08:29:07.792004579Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R8wjaedoiarBcdJxSdsNG9i","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.797167_75.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.11","timestamp":"2020-06-04T08:29:07.797167708Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R8wjaedoiarBcdJxSdsNG9i","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.798799_76.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:07.798799372Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFqu-tIqXwMndR9GZsMEdUg","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.801827_77.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:07.801827701Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFqu-tIqXwMndR9GZsMEdUg","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259347.895642_78.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.12","timestamp":"2020-06-04T08:29:07.895642957Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_813CHCsdbVJhG-8Friwea0Q","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.900475_79.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.12","timestamp":"2020-06-04T08:29:07.900475349Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_813CHCsdbVJhG-8Friwea0Q","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259347.903751_80.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:07.903751639Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4jTRpPwqpn_U-wRoBFCv-5t","ver":1},"status":{"status":"write"}}},{"id":"1_1591259347.907394_81.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:07.907394318Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_4jTRpPwqpn_U-wRoBFCv-5t","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259348.003913_82.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.13","timestamp":"2020-06-04T08:29:08.003913702Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vjl-KiOIIS-mHj4Knkvyvbn","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.008410_83.1","section":"bucket.instance","name":"gpxthc-4:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.13","timestamp":"2020-06-04T08:29:08.008410230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vjl-KiOIIS-mHj4Knkvyvbn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259348.010405_84.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:08.010405065Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YXdPZK993bwZgNRYKqmPKbg","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.013902_85.1","section":"bucket","name":"gpxthc-4","timestamp":"2020-06-04T08:29:08.013902468Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YXdPZK993bwZgNRYKqmPKbg","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259348.061824_86.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.14","timestamp":"2020-06-04T08:29:08.061824583Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GmPkZfz9onYDYux9NvroVpb","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.066453_87.1","section":"bucket.instance","name":"gpxthc-5:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.14","timestamp":"2020-06-04T08:29:08.066453149Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GmPkZfz9onYDYux9NvroVpb","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259348.068278_88.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:08.068278859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n5UFL0MW6wtQDqqBX5p9xOv","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.071562_89.1","section":"bucket","name":"gpxthc-5","timestamp":"2020-06-04T08:29:08.071562898Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_n5UFL0MW6wtQDqqBX5p9xOv","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259348.164678_90.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.15","timestamp":"2020-06-04T08:29:08.164678779Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wnNQvYf5A6zQTv_L1L16Vj_","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.169209_91.1","section":"bucket.instance","name":"gpxthc-6:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.15","timestamp":"2020-06-04T08:29:08.169209509Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wnNQvYf5A6zQTv_L1L16Vj_","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259348.171482_92.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:08.171482220Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HuczynHZJw9zrk53biUsBon","ver":1},"status":{"status":"write"}}},{"id":"1_1591259348.175359_93.1","section":"bucket","name":"gpxthc-6","timestamp":"2020-06-04T08:29:08.175359125Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HuczynHZJw9zrk53biUsBon","ver":1},"status":{"status":"abort"}}},{"id":"1_1591259349.444497_94.1","section":"bucket.instance","name":"gpxthc-7:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.16","timestamp":"2020-06-04T08:29:09.444497949Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0hH3KdNQ7Ror_-PbjMomrOK","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.449423_95.1","section":"bucket.instance","name":"gpxthc-7:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.16","timestamp":"2020-06-04T08:29:09.449423359Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0hH3KdNQ7Ror_-PbjMomrOK","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259349.451676_96.1","section":"bucket","name":"gpxthc-7","timestamp":"2020-06-04T08:29:09.451676822Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.455775_97.1","section":"bucket","name":"gpxthc-7","timestamp":"2020-06-04T08:29:09.455775359Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259349.478408_98.1","section":"bucket.instance","name":"gpxthc-8:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.17","timestamp":"2020-06-04T08:29:09.478408677Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3tdK5SOwUxP7sixtd5Iiusu","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.483441_99.1","section":"bucket.instance","name":"gpxthc-8:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.17","timestamp":"2020-06-04T08:29:09.483441043Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3tdK5SOwUxP7sixtd5Iiusu","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259349.485210_100.1","section":"bucket","name":"gpxthc-8","timestamp":"2020-06-04T08:29:09.485210362Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.489008_101.1","section":"bucket","name":"gpxthc-8","timestamp":"2020-06-04T08:29:09.489008069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259349.542500_102.1","section":"bucket.instance","name":"gpxthc-9:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.18","timestamp":"2020-06-04T08:29:09.542500556Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZM4c0G2TMLKguEZQHJhJ_U1","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.547391_103.1","section":"bucket.instance","name":"gpxthc-9:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.18","timestamp":"2020-06-04T08:29:09.547391664Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZM4c0G2TMLKguEZQHJhJ_U1","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259349.549662_104.1","section":"bucket","name":"gpxthc-9","timestamp":"2020-06-04T08:29:09.549662472Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":1},"status":{"status":"write"}}},{"id":"1_1591259349.557874_105.1","section":"bucket","name":"gpxthc-9","timestamp":"2020-06-04T08:29:09.557874857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":1},"status":{"status":"complete"}}},{"id":"1_1591259350.809493_106.1","section":"bucket","name":"gpxthc-7","timestamp":"2020-06-04T08:29:10.809493171Z","data":{"read_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":1},"write_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":2},"status":{"status":"remove"}}},{"id":"1_1591259350.815979_107.1","section":"bucket","name":"gpxthc-7","timestamp":"2020-06-04T08:29:10.815979805Z","data":{"read_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":1},"write_version":{"tag":"_cecCIVFZsnZteB3_f6Rp1jP","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259350.882584_108.1","section":"bucket","name":"gpxthc-8","timestamp":"2020-06-04T08:29:10.882584350Z","data":{"read_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":1},"write_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":2},"status":{"status":"remove"}}},{"id":"1_1591259350.890059_109.1","section":"bucket","name":"gpxthc-8","timestamp":"2020-06-04T08:29:10.890059979Z","data":{"read_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":1},"write_version":{"tag":"_JkHvvZwS5CeSbkcwaPr8jpo","ver":2},"status":{"status":"complete"}}},{"id":"1_1591259350.978944_110.1","section":"bucket","name":"gpxthc-9","timestamp":"2020-06-04T08:29:10.978944722Z","data":{"read_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":1},"write_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":2},"status":{"status":"remove"}}},{"id":"1_1591259350.985454_111.1","section":"bucket","name":"gpxthc-9","timestamp":"2020-06-04T08:29:10.985454315Z","data":{"read_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":1},"write_version":{"tag":"_Sq7g4CbUObmArC7oDrGW5bj","ver":2},"status":{"status":"complete"}}}] 2020-06-04T08:41:14.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '8681f463-3379-4ae0-aef4-6adb4b0f951f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:14.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '8681f463-3379-4ae0-aef4-6adb4b0f951f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:14.076 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 8681f463-3379-4ae0-aef4-6adb4b0f951f --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:14.118 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:14.118 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:14.364 INFO:teuthology.orchestra.run.smithi060.stdout:[{"id":"1_1591260003.725880_100.1","section":"bucket.instance","name":"gpxthc-42:d7df857b-1e19-4b4c-aa22-43c42d072194.6610.1","timestamp":"2020-06-04T08:40:03.725880236Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cDro1Oq5D7wDQezjOm3mRoo","ver":1},"status":{"status":"write"}}},{"id":"1_1591260003.730231_101.1","section":"bucket.instance","name":"gpxthc-42:d7df857b-1e19-4b4c-aa22-43c42d072194.6610.1","timestamp":"2020-06-04T08:40:03.730231931Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cDro1Oq5D7wDQezjOm3mRoo","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260003.732466_102.1","section":"bucket","name":"gpxthc-42","timestamp":"2020-06-04T08:40:03.732466037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9jCoCArkMLSqlFNIdtB211u","ver":1},"status":{"status":"write"}}},{"id":"1_1591260003.736486_103.1","section":"bucket","name":"gpxthc-42","timestamp":"2020-06-04T08:40:03.736486317Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9jCoCArkMLSqlFNIdtB211u","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260003.783018_104.1","section":"bucket.instance","name":"gpxthc-43:d7df857b-1e19-4b4c-aa22-43c42d072194.6610.2","timestamp":"2020-06-04T08:40:03.783018579Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZIursW0OZVgH8q4fLpvUDLN","ver":1},"status":{"status":"write"}}},{"id":"1_1591260003.787771_105.1","section":"bucket.instance","name":"gpxthc-43:d7df857b-1e19-4b4c-aa22-43c42d072194.6610.2","timestamp":"2020-06-04T08:40:03.787771437Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZIursW0OZVgH8q4fLpvUDLN","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260003.789699_106.1","section":"bucket","name":"gpxthc-43","timestamp":"2020-06-04T08:40:03.789699781Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DK4OfCmdyNmYQFGLhMQroBK","ver":1},"status":{"status":"write"}}},{"id":"1_1591260003.794286_107.1","section":"bucket","name":"gpxthc-43","timestamp":"2020-06-04T08:40:03.794286997Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DK4OfCmdyNmYQFGLhMQroBK","ver":1},"status":{"status":"complete"}}}] 2020-06-04T08:41:14.364 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '8681f463-3379-4ae0-aef4-6adb4b0f951f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:14.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '8681f463-3379-4ae0-aef4-6adb4b0f951f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:14.378 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 8681f463-3379-4ae0-aef4-6adb4b0f951f --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:14.418 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:14.418 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:14.651 INFO:teuthology.orchestra.run.smithi196.stdout:[{"id":"1_1591260003.725880_100.1","section":"bucket.instance","name":"gpxthc-42:d7df857b-1e19-4b4c-aa22-43c42d072194.6610.1","timestamp":"2020-06-04T08:40:03.725880236Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cDro1Oq5D7wDQezjOm3mRoo","ver":1},"status":{"status":"write"}}},{"id":"1_1591260003.730231_101.1","section":"bucket.instance","name":"gpxthc-42:d7df857b-1e19-4b4c-aa22-43c42d072194.6610.1","timestamp":"2020-06-04T08:40:03.730231931Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cDro1Oq5D7wDQezjOm3mRoo","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260003.732466_102.1","section":"bucket","name":"gpxthc-42","timestamp":"2020-06-04T08:40:03.732466037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9jCoCArkMLSqlFNIdtB211u","ver":1},"status":{"status":"write"}}},{"id":"1_1591260003.736486_103.1","section":"bucket","name":"gpxthc-42","timestamp":"2020-06-04T08:40:03.736486317Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9jCoCArkMLSqlFNIdtB211u","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260003.783018_104.1","section":"bucket.instance","name":"gpxthc-43:d7df857b-1e19-4b4c-aa22-43c42d072194.6610.2","timestamp":"2020-06-04T08:40:03.783018579Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZIursW0OZVgH8q4fLpvUDLN","ver":1},"status":{"status":"write"}}},{"id":"1_1591260003.787771_105.1","section":"bucket.instance","name":"gpxthc-43:d7df857b-1e19-4b4c-aa22-43c42d072194.6610.2","timestamp":"2020-06-04T08:40:03.787771437Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZIursW0OZVgH8q4fLpvUDLN","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260003.789699_106.1","section":"bucket","name":"gpxthc-43","timestamp":"2020-06-04T08:40:03.789699781Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DK4OfCmdyNmYQFGLhMQroBK","ver":1},"status":{"status":"write"}}},{"id":"1_1591260003.794286_107.1","section":"bucket","name":"gpxthc-43","timestamp":"2020-06-04T08:40:03.794286997Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DK4OfCmdyNmYQFGLhMQroBK","ver":1},"status":{"status":"complete"}}}] 2020-06-04T08:41:14.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '8681f463-3379-4ae0-aef4-6adb4b0f951f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:14.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '8681f463-3379-4ae0-aef4-6adb4b0f951f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:14.653 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 8681f463-3379-4ae0-aef4-6adb4b0f951f --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:14.693 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:14.694 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:14.942 INFO:teuthology.orchestra.run.smithi060.stdout:[{"id":"1_1591260003.725880_100.1","section":"bucket.instance","name":"gpxthc-42:d7df857b-1e19-4b4c-aa22-43c42d072194.6610.1","timestamp":"2020-06-04T08:40:03.725880236Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cDro1Oq5D7wDQezjOm3mRoo","ver":1},"status":{"status":"write"}}},{"id":"1_1591260003.730231_101.1","section":"bucket.instance","name":"gpxthc-42:d7df857b-1e19-4b4c-aa22-43c42d072194.6610.1","timestamp":"2020-06-04T08:40:03.730231931Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cDro1Oq5D7wDQezjOm3mRoo","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260003.732466_102.1","section":"bucket","name":"gpxthc-42","timestamp":"2020-06-04T08:40:03.732466037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9jCoCArkMLSqlFNIdtB211u","ver":1},"status":{"status":"write"}}},{"id":"1_1591260003.736486_103.1","section":"bucket","name":"gpxthc-42","timestamp":"2020-06-04T08:40:03.736486317Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9jCoCArkMLSqlFNIdtB211u","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260003.783018_104.1","section":"bucket.instance","name":"gpxthc-43:d7df857b-1e19-4b4c-aa22-43c42d072194.6610.2","timestamp":"2020-06-04T08:40:03.783018579Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZIursW0OZVgH8q4fLpvUDLN","ver":1},"status":{"status":"write"}}},{"id":"1_1591260003.787771_105.1","section":"bucket.instance","name":"gpxthc-43:d7df857b-1e19-4b4c-aa22-43c42d072194.6610.2","timestamp":"2020-06-04T08:40:03.787771437Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZIursW0OZVgH8q4fLpvUDLN","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260003.789699_106.1","section":"bucket","name":"gpxthc-43","timestamp":"2020-06-04T08:40:03.789699781Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DK4OfCmdyNmYQFGLhMQroBK","ver":1},"status":{"status":"write"}}},{"id":"1_1591260003.794286_107.1","section":"bucket","name":"gpxthc-43","timestamp":"2020-06-04T08:40:03.794286997Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DK4OfCmdyNmYQFGLhMQroBK","ver":1},"status":{"status":"complete"}}}] 2020-06-04T08:41:14.943 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '8681f463-3379-4ae0-aef4-6adb4b0f951f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:14.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '8681f463-3379-4ae0-aef4-6adb4b0f951f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:14.945 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 8681f463-3379-4ae0-aef4-6adb4b0f951f --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:14.983 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:14.984 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:15.211 INFO:teuthology.orchestra.run.smithi196.stdout:[{"id":"1_1591260003.725880_100.1","section":"bucket.instance","name":"gpxthc-42:d7df857b-1e19-4b4c-aa22-43c42d072194.6610.1","timestamp":"2020-06-04T08:40:03.725880236Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cDro1Oq5D7wDQezjOm3mRoo","ver":1},"status":{"status":"write"}}},{"id":"1_1591260003.730231_101.1","section":"bucket.instance","name":"gpxthc-42:d7df857b-1e19-4b4c-aa22-43c42d072194.6610.1","timestamp":"2020-06-04T08:40:03.730231931Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cDro1Oq5D7wDQezjOm3mRoo","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260003.732466_102.1","section":"bucket","name":"gpxthc-42","timestamp":"2020-06-04T08:40:03.732466037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9jCoCArkMLSqlFNIdtB211u","ver":1},"status":{"status":"write"}}},{"id":"1_1591260003.736486_103.1","section":"bucket","name":"gpxthc-42","timestamp":"2020-06-04T08:40:03.736486317Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9jCoCArkMLSqlFNIdtB211u","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260003.783018_104.1","section":"bucket.instance","name":"gpxthc-43:d7df857b-1e19-4b4c-aa22-43c42d072194.6610.2","timestamp":"2020-06-04T08:40:03.783018579Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZIursW0OZVgH8q4fLpvUDLN","ver":1},"status":{"status":"write"}}},{"id":"1_1591260003.787771_105.1","section":"bucket.instance","name":"gpxthc-43:d7df857b-1e19-4b4c-aa22-43c42d072194.6610.2","timestamp":"2020-06-04T08:40:03.787771437Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZIursW0OZVgH8q4fLpvUDLN","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260003.789699_106.1","section":"bucket","name":"gpxthc-43","timestamp":"2020-06-04T08:40:03.789699781Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DK4OfCmdyNmYQFGLhMQroBK","ver":1},"status":{"status":"write"}}},{"id":"1_1591260003.794286_107.1","section":"bucket","name":"gpxthc-43","timestamp":"2020-06-04T08:40:03.794286997Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DK4OfCmdyNmYQFGLhMQroBK","ver":1},"status":{"status":"complete"}}}] 2020-06-04T08:41:15.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '387a2dc2-99d4-41c4-ad70-50eff9e4eab1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:15.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '387a2dc2-99d4-41c4-ad70-50eff9e4eab1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:15.213 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 387a2dc2-99d4-41c4-ad70-50eff9e4eab1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:15.254 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:15.255 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:15.496 INFO:teuthology.orchestra.run.smithi060.stdout:[{"id":"1_1591260064.871889_81.1","section":"bucket.instance","name":"gpxthc-44:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.1","timestamp":"2020-06-04T08:41:04.871889637Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mpyBrOKR8obdUdp9MhaoJtY","ver":1},"status":{"status":"write"}}},{"id":"1_1591260064.877694_87.1","section":"bucket.instance","name":"gpxthc-44:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.1","timestamp":"2020-06-04T08:41:04.877694905Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mpyBrOKR8obdUdp9MhaoJtY","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260064.879648_88.1","section":"bucket","name":"gpxthc-44","timestamp":"2020-06-04T08:41:04.879648612Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dIZfPd8jiMes5AoUgvzUqcn","ver":1},"status":{"status":"write"}}},{"id":"1_1591260064.884139_89.1","section":"bucket","name":"gpxthc-44","timestamp":"2020-06-04T08:41:04.884139951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dIZfPd8jiMes5AoUgvzUqcn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260064.926804_90.1","section":"bucket.instance","name":"gpxthc-45:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.2","timestamp":"2020-06-04T08:41:04.926804677Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-ap3fwmn4BauismQiu95Bft","ver":1},"status":{"status":"write"}}},{"id":"1_1591260064.931616_91.1","section":"bucket.instance","name":"gpxthc-45:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.2","timestamp":"2020-06-04T08:41:04.931616334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-ap3fwmn4BauismQiu95Bft","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260064.933561_92.1","section":"bucket","name":"gpxthc-45","timestamp":"2020-06-04T08:41:04.933561093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ebLUAbjEBH9_TZuKisJ96mi","ver":1},"status":{"status":"write"}}},{"id":"1_1591260064.937745_93.1","section":"bucket","name":"gpxthc-45","timestamp":"2020-06-04T08:41:04.937745864Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ebLUAbjEBH9_TZuKisJ96mi","ver":1},"status":{"status":"complete"}}}] 2020-06-04T08:41:15.497 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '387a2dc2-99d4-41c4-ad70-50eff9e4eab1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:15.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '387a2dc2-99d4-41c4-ad70-50eff9e4eab1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:15.498 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 387a2dc2-99d4-41c4-ad70-50eff9e4eab1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:15.536 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:15.537 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:15.769 INFO:teuthology.orchestra.run.smithi196.stdout:[{"id":"1_1591260064.871889_81.1","section":"bucket.instance","name":"gpxthc-44:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.1","timestamp":"2020-06-04T08:41:04.871889637Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mpyBrOKR8obdUdp9MhaoJtY","ver":1},"status":{"status":"write"}}},{"id":"1_1591260064.877694_87.1","section":"bucket.instance","name":"gpxthc-44:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.1","timestamp":"2020-06-04T08:41:04.877694905Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mpyBrOKR8obdUdp9MhaoJtY","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260064.879648_88.1","section":"bucket","name":"gpxthc-44","timestamp":"2020-06-04T08:41:04.879648612Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dIZfPd8jiMes5AoUgvzUqcn","ver":1},"status":{"status":"write"}}},{"id":"1_1591260064.884139_89.1","section":"bucket","name":"gpxthc-44","timestamp":"2020-06-04T08:41:04.884139951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dIZfPd8jiMes5AoUgvzUqcn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260064.926804_90.1","section":"bucket.instance","name":"gpxthc-45:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.2","timestamp":"2020-06-04T08:41:04.926804677Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-ap3fwmn4BauismQiu95Bft","ver":1},"status":{"status":"write"}}},{"id":"1_1591260064.931616_91.1","section":"bucket.instance","name":"gpxthc-45:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.2","timestamp":"2020-06-04T08:41:04.931616334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-ap3fwmn4BauismQiu95Bft","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260064.933561_92.1","section":"bucket","name":"gpxthc-45","timestamp":"2020-06-04T08:41:04.933561093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ebLUAbjEBH9_TZuKisJ96mi","ver":1},"status":{"status":"write"}}},{"id":"1_1591260064.937745_93.1","section":"bucket","name":"gpxthc-45","timestamp":"2020-06-04T08:41:04.937745864Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ebLUAbjEBH9_TZuKisJ96mi","ver":1},"status":{"status":"complete"}}}] 2020-06-04T08:41:15.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '387a2dc2-99d4-41c4-ad70-50eff9e4eab1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:15.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '387a2dc2-99d4-41c4-ad70-50eff9e4eab1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:15.771 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 387a2dc2-99d4-41c4-ad70-50eff9e4eab1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:15.810 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:15.810 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:16.046 INFO:teuthology.orchestra.run.smithi060.stdout:[{"id":"1_1591260064.871889_81.1","section":"bucket.instance","name":"gpxthc-44:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.1","timestamp":"2020-06-04T08:41:04.871889637Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mpyBrOKR8obdUdp9MhaoJtY","ver":1},"status":{"status":"write"}}},{"id":"1_1591260064.877694_87.1","section":"bucket.instance","name":"gpxthc-44:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.1","timestamp":"2020-06-04T08:41:04.877694905Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mpyBrOKR8obdUdp9MhaoJtY","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260064.879648_88.1","section":"bucket","name":"gpxthc-44","timestamp":"2020-06-04T08:41:04.879648612Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dIZfPd8jiMes5AoUgvzUqcn","ver":1},"status":{"status":"write"}}},{"id":"1_1591260064.884139_89.1","section":"bucket","name":"gpxthc-44","timestamp":"2020-06-04T08:41:04.884139951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dIZfPd8jiMes5AoUgvzUqcn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260064.926804_90.1","section":"bucket.instance","name":"gpxthc-45:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.2","timestamp":"2020-06-04T08:41:04.926804677Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-ap3fwmn4BauismQiu95Bft","ver":1},"status":{"status":"write"}}},{"id":"1_1591260064.931616_91.1","section":"bucket.instance","name":"gpxthc-45:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.2","timestamp":"2020-06-04T08:41:04.931616334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-ap3fwmn4BauismQiu95Bft","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260064.933561_92.1","section":"bucket","name":"gpxthc-45","timestamp":"2020-06-04T08:41:04.933561093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ebLUAbjEBH9_TZuKisJ96mi","ver":1},"status":{"status":"write"}}},{"id":"1_1591260064.937745_93.1","section":"bucket","name":"gpxthc-45","timestamp":"2020-06-04T08:41:04.937745864Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ebLUAbjEBH9_TZuKisJ96mi","ver":1},"status":{"status":"complete"}}}] 2020-06-04T08:41:16.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '387a2dc2-99d4-41c4-ad70-50eff9e4eab1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:16.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '387a2dc2-99d4-41c4-ad70-50eff9e4eab1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:16.056 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 387a2dc2-99d4-41c4-ad70-50eff9e4eab1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:16.097 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:16.097 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:16.345 INFO:teuthology.orchestra.run.smithi196.stdout:[{"id":"1_1591260064.871889_81.1","section":"bucket.instance","name":"gpxthc-44:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.1","timestamp":"2020-06-04T08:41:04.871889637Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mpyBrOKR8obdUdp9MhaoJtY","ver":1},"status":{"status":"write"}}},{"id":"1_1591260064.877694_87.1","section":"bucket.instance","name":"gpxthc-44:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.1","timestamp":"2020-06-04T08:41:04.877694905Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_mpyBrOKR8obdUdp9MhaoJtY","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260064.879648_88.1","section":"bucket","name":"gpxthc-44","timestamp":"2020-06-04T08:41:04.879648612Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dIZfPd8jiMes5AoUgvzUqcn","ver":1},"status":{"status":"write"}}},{"id":"1_1591260064.884139_89.1","section":"bucket","name":"gpxthc-44","timestamp":"2020-06-04T08:41:04.884139951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dIZfPd8jiMes5AoUgvzUqcn","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260064.926804_90.1","section":"bucket.instance","name":"gpxthc-45:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.2","timestamp":"2020-06-04T08:41:04.926804677Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-ap3fwmn4BauismQiu95Bft","ver":1},"status":{"status":"write"}}},{"id":"1_1591260064.931616_91.1","section":"bucket.instance","name":"gpxthc-45:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.2","timestamp":"2020-06-04T08:41:04.931616334Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-ap3fwmn4BauismQiu95Bft","ver":1},"status":{"status":"complete"}}},{"id":"1_1591260064.933561_92.1","section":"bucket","name":"gpxthc-45","timestamp":"2020-06-04T08:41:04.933561093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ebLUAbjEBH9_TZuKisJ96mi","ver":1},"status":{"status":"write"}}},{"id":"1_1591260064.937745_93.1","section":"bucket","name":"gpxthc-45","timestamp":"2020-06-04T08:41:04.937745864Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ebLUAbjEBH9_TZuKisJ96mi","ver":1},"status":{"status":"complete"}}}] 2020-06-04T08:41:16.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:41:16.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:41:16.348 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:41:16.384 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:16.384 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:16.675 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:41:16.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:41:16.677 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:41:16.714 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:16.714 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:17.128 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:41:17.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:41:17.130 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:41:17.169 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:17.170 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:17.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:41:17.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:41:17.429 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:41:17.465 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:17.465 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:17.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '838ed6b2-fa2f-4917-a5c1-6c82bb405cda', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:17.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '838ed6b2-fa2f-4917-a5c1-6c82bb405cda', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:17.852 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 838ed6b2-fa2f-4917-a5c1-6c82bb405cda --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:17.891 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:17.892 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:18.138 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2020-06-04T08:41:18.139 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '838ed6b2-fa2f-4917-a5c1-6c82bb405cda', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:18.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '838ed6b2-fa2f-4917-a5c1-6c82bb405cda', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:18.141 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 838ed6b2-fa2f-4917-a5c1-6c82bb405cda --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:18.189 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:18.189 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:18.414 INFO:teuthology.orchestra.run.smithi196.stdout:[] 2020-06-04T08:41:18.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '838ed6b2-fa2f-4917-a5c1-6c82bb405cda', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:18.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '838ed6b2-fa2f-4917-a5c1-6c82bb405cda', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:18.415 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 838ed6b2-fa2f-4917-a5c1-6c82bb405cda --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:18.456 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:18.456 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:18.695 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2020-06-04T08:41:18.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '838ed6b2-fa2f-4917-a5c1-6c82bb405cda', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:18.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '838ed6b2-fa2f-4917-a5c1-6c82bb405cda', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:18.696 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 838ed6b2-fa2f-4917-a5c1-6c82bb405cda --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:18.736 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:18.737 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:18.969 INFO:teuthology.orchestra.run.smithi196.stdout:[] 2020-06-04T08:41:18.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '8681f463-3379-4ae0-aef4-6adb4b0f951f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:18.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '8681f463-3379-4ae0-aef4-6adb4b0f951f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:18.971 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 8681f463-3379-4ae0-aef4-6adb4b0f951f --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:19.010 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:19.010 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:19.260 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2020-06-04T08:41:19.261 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '8681f463-3379-4ae0-aef4-6adb4b0f951f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:19.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '8681f463-3379-4ae0-aef4-6adb4b0f951f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:19.262 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 8681f463-3379-4ae0-aef4-6adb4b0f951f --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:19.301 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:19.302 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:19.535 INFO:teuthology.orchestra.run.smithi196.stdout:[] 2020-06-04T08:41:19.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '8681f463-3379-4ae0-aef4-6adb4b0f951f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:19.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '8681f463-3379-4ae0-aef4-6adb4b0f951f', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:19.537 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 8681f463-3379-4ae0-aef4-6adb4b0f951f --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:19.578 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:19.578 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:19.817 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2020-06-04T08:41:19.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '8681f463-3379-4ae0-aef4-6adb4b0f951f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:19.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '8681f463-3379-4ae0-aef4-6adb4b0f951f', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:19.819 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 8681f463-3379-4ae0-aef4-6adb4b0f951f --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:19.858 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:19.858 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:20.100 INFO:teuthology.orchestra.run.smithi196.stdout:[] 2020-06-04T08:41:20.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '387a2dc2-99d4-41c4-ad70-50eff9e4eab1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:20.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '387a2dc2-99d4-41c4-ad70-50eff9e4eab1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:20.102 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 387a2dc2-99d4-41c4-ad70-50eff9e4eab1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:20.141 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:20.142 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:20.393 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2020-06-04T08:41:20.393 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '387a2dc2-99d4-41c4-ad70-50eff9e4eab1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:20.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '387a2dc2-99d4-41c4-ad70-50eff9e4eab1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:20.395 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 387a2dc2-99d4-41c4-ad70-50eff9e4eab1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:20.435 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:20.435 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:20.668 INFO:teuthology.orchestra.run.smithi196.stdout:[] 2020-06-04T08:41:20.669 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '387a2dc2-99d4-41c4-ad70-50eff9e4eab1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:20.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '387a2dc2-99d4-41c4-ad70-50eff9e4eab1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:20.671 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 387a2dc2-99d4-41c4-ad70-50eff9e4eab1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:20.710 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:20.710 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:20.954 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2020-06-04T08:41:20.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '387a2dc2-99d4-41c4-ad70-50eff9e4eab1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:20.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '387a2dc2-99d4-41c4-ad70-50eff9e4eab1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:20.956 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 387a2dc2-99d4-41c4-ad70-50eff9e4eab1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:20.998 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:20.998 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:21.230 INFO:teuthology.orchestra.run.smithi196.stdout:[] 2020-06-04T08:41:21.231 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_period_incremental_sync ... ok 2020-06-04T08:41:21.232 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-46 2020-06-04T08:41:21.276 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gpxthc-47 2020-06-04T08:41:21.349 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gpxthc-48 2020-06-04T08:41:21.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:21.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:21.576 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:21.622 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:21.623 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:21.829 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259652.395954_88.1","last_update":"2020-06-04T08:34:12.395954Z"},{"marker":"1_1591259930.643002_247.1","last_update":"2020-06-04T08:38:50.643002Z"},{"marker":"1_1591259589.882369_86.1","last_update":"2020-06-04T08:33:09.882369Z"},{"marker":"1_1591260081.423742_296.1","last_update":"2020-06-04T08:41:21.423742Z"}] 2020-06-04T08:41:21.830 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-06-04T08:41:21.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:21.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:21.832 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:21.873 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:21.873 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:22.087 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11207024744551397021},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591259652.395954_88.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:34:12.395954Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260081.423742_296.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:21.423742Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:41:22.087 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-06-04T08:41:22.088 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:22.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:22.090 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:22.132 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:22.132 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:22.362 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259652.395954_88.1","last_update":"2020-06-04T08:34:12.395954Z"},{"marker":"1_1591259930.643002_247.1","last_update":"2020-06-04T08:38:50.643002Z"},{"marker":"1_1591259589.882369_86.1","last_update":"2020-06-04T08:33:09.882369Z"},{"marker":"1_1591260081.810624_312.1","last_update":"2020-06-04T08:41:21.810624Z"}] 2020-06-04T08:41:22.363 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-06-04T08:41:22.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:22.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:22.366 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:22.450 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:22.450 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:22.677 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2627143167498148538},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591259652.395954_88.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:34:12.395954Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260081.810624_312.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:21.810624Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:41:22.678 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-06-04T08:41:22.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:22.679 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:22.680 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:22.754 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:22.755 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:22.996 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259652.395954_88.1","last_update":"2020-06-04T08:34:12.395954Z"},{"marker":"1_1591259930.643002_247.1","last_update":"2020-06-04T08:38:50.643002Z"},{"marker":"1_1591259589.882369_86.1","last_update":"2020-06-04T08:33:09.882369Z"},{"marker":"1_1591260081.810624_312.1","last_update":"2020-06-04T08:41:21.810624Z"}] 2020-06-04T08:41:22.996 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-06-04T08:41:22.997 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:22.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:22.998 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:23.044 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:23.045 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:23.313 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7644369918941386260},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591259652.395954_88.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:34:12.395954Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260081.810624_312.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:21.810624Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:41:23.314 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-06-04T08:41:23.315 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:23.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:23.316 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:23.397 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:23.398 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:23.624 INFO:teuthology.orchestra.run.smithi196.stdout:[{"marker":"1_1591259652.539635_76.1","last_update":"2020-06-04T08:34:12.539635Z"},{"marker":"1_1591259930.639528_257.1","last_update":"2020-06-04T08:38:50.639528Z"},{"marker":"1_1591259589.701718_107.1","last_update":"2020-06-04T08:33:09.701718Z"},{"marker":"1_1591260081.759349_309.1","last_update":"2020-06-04T08:41:21.759349Z"}] 2020-06-04T08:41:23.625 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-06-04T08:41:23.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:23.626 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:23.626 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:23.671 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:23.672 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:23.915 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1308327179334993417},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591259652.539635_76.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:34:12.539635Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.639528_257.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.639528Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.701718_107.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.701718Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260081.759349_309.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:21.759349Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:41:23.916 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-06-04T08:41:23.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:23.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:23.919 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:23.960 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:23.961 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:24.173 INFO:teuthology.orchestra.run.smithi196.stdout:[{"marker":"1_1591259652.539635_76.1","last_update":"2020-06-04T08:34:12.539635Z"},{"marker":"1_1591259930.639528_257.1","last_update":"2020-06-04T08:38:50.639528Z"},{"marker":"1_1591259589.701718_107.1","last_update":"2020-06-04T08:33:09.701718Z"},{"marker":"1_1591260081.759349_309.1","last_update":"2020-06-04T08:41:21.759349Z"}] 2020-06-04T08:41:24.173 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-06-04T08:41:24.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:24.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:24.179 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:24.221 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:24.223 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:24.455 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":16507416453976119466},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591259652.539635_76.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:34:12.539635Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.639528_257.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.639528Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.701718_107.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.701718Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260081.759349_309.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:21.759349Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:41:24.456 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-06-04T08:41:24.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:24.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:24.457 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:24.540 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:24.541 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:24.764 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259652.444513_63.1","last_update":"2020-06-04T08:34:12.444513Z"},{"marker":"1_1591259930.572137_196.1","last_update":"2020-06-04T08:38:50.572137Z"},{"marker":"1_1591259589.886327_75.1","last_update":"2020-06-04T08:33:09.886327Z"},{"marker":"1_1591260081.809255_183.1","last_update":"2020-06-04T08:41:21.809255Z"}] 2020-06-04T08:41:24.764 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-06-04T08:41:24.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:24.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:24.766 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:24.849 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:24.850 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:25.081 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3830535810264008500},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591259652.444513_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:34:12.444513Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.572137_196.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.572137Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.886327_75.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.886327Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591259916.484392_156.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:36.484392Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:41:25.082 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone3: shard=3 master=1_1591260081.809255_183.1 target=1_1591259916.484392_156.3 2020-06-04T08:41:30.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:30.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:30.086 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:30.130 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:30.130 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:30.378 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3830535810264008500},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591259652.444513_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:34:12.444513Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.572137_196.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.572137Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.886327_75.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.886327Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260081.809255_183.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:21.809255Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:41:30.379 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-06-04T08:41:30.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:30.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:30.381 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:30.467 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:30.467 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:30.709 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591259652.444513_63.1","last_update":"2020-06-04T08:34:12.444513Z"},{"marker":"1_1591259930.572137_196.1","last_update":"2020-06-04T08:38:50.572137Z"},{"marker":"1_1591259589.886327_75.1","last_update":"2020-06-04T08:33:09.886327Z"},{"marker":"1_1591260081.809255_183.1","last_update":"2020-06-04T08:41:21.809255Z"}] 2020-06-04T08:41:30.710 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-06-04T08:41:30.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:30.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:30.712 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:30.759 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:30.759 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:30.982 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9078143951431167771},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591259652.444513_63.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:34:12.444513Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.572137_196.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.572137Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.886327_75.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.886327Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260081.809255_183.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:21.809255Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:41:30.983 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-06-04T08:41:30.984 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:41:30.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:41:30.985 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:41:31.020 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:31.020 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:31.281 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:31.282 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:31.283 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:31.365 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:31.365 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:31.611 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2020-06-04T08:41:31.612 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:41:31.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:41:31.613 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:41:31.651 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:31.652 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:31.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:31.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:31.872 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:31.950 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:31.951 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:32.187 INFO:teuthology.orchestra.run.smithi196.stdout:[] 2020-06-04T08:41:32.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:41:32.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:41:32.189 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:41:32.234 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:32.234 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:32.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:32.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:41:32.476 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:41:32.556 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:32.556 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:32.808 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2020-06-04T08:41:32.809 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_autotrim ... ok 2020-06-04T08:41:32.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:41:32.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:41:32.812 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:41:32.851 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:32.852 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:32.880 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:41:32.876+0000 7f2b23bc10c0 0 failed reading obj info from .rgw.root:zone_info.463a7035-f6f9-4bfc-a894-c421122fd949: (2) No such file or directory 2020-06-04T08:41:32.881 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:41:32.876+0000 7f2b23bc10c0 0 WARNING: could not read zone params for zone id=463a7035-f6f9-4bfc-a894-c421122fd949 name=test-zone3 2020-06-04T08:41:32.881 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:41:32.876+0000 7f2b23bc10c0 0 failed reading obj info from .rgw.root:zone_info.7ad4cc51-2d19-48f1-a21e-ef15faf62b94: (2) No such file or directory 2020-06-04T08:41:32.882 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:41:32.876+0000 7f2b23bc10c0 0 WARNING: could not read zone params for zone id=7ad4cc51-2d19-48f1-a21e-ef15faf62b94 name=test-zone1 2020-06-04T08:41:32.897 INFO:teuthology.orchestra.run.smithi196.stdout:{"id":"d7df857b-1e19-4b4c-aa22-43c42d072194","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"M2KEYTP439P9KXXD","secret_key":"Ad56bMSsbqPkjfMTVush7dBgYdLf27ul"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","notif_pool":"test-zone2.rgw.log:notif"} 2020-06-04T08:41:32.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:41:32.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:41:32.900 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:41:32.975 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:41:32.976 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:41:33.207 INFO:teuthology.orchestra.run.smithi196.stderr:Sending period to new master zone 7ad4cc51-2d19-48f1-a21e-ef15faf62b94 2020-06-04T08:41:33.302 INFO:teuthology.orchestra.run.smithi196.stdout:{"id":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","epoch":2,"predecessor_uuid":"8681f463-3379-4ae0-aef4-6adb4b0f951f","sync_status":["1_1591260003.794286_107.1","","",""],"period_map":{"id":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","zones":[{"id":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","name":"test-zone4","endpoints":["http://smithi196.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"463a7035-f6f9-4bfc-a894-c421122fd949","name":"test-zone3","endpoints":["http://smithi060.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","name":"test-zone1","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7df857b-1e19-4b4c-aa22-43c42d072194","name":"test-zone2","endpoints":["http://smithi196.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","val":2528435237},{"key":"463a7035-f6f9-4bfc-a894-c421122fd949","val":2270195151},{"key":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","val":3611466229},{"key":"d7df857b-1e19-4b4c-aa22-43c42d072194","val":443345077}]},"master_zonegroup":"test-zonegroup","master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","realm_name":"test-realm","realm_epoch":4} 2020-06-04T08:41:33.303 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2020-06-04T08:42:33.304 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-49 2020-06-04T08:42:33.403 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:42:33.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:42:33.405 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:42:33.452 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:42:33.452 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:42:33.653 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=387a2dc2-99d4-41c4-ad70-50eff9e4eab1 2020-06-04T08:42:33.695 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260153.347537_146.1","last_update":"2020-06-04T08:42:33.347537Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:42:33.696 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:42:33.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:42:33.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:42:33.698 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:42:33.740 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:42:33.740 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:42:33.964 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260081.388568_132.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:41:21.388568Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:42:33.965 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591260153.347537_146.1 target=1_1591260081.388568_132.1 2020-06-04T08:42:38.968 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:42:38.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:42:38.970 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:42:39.011 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:42:39.012 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:42:39.235 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260081.388568_132.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:41:21.388568Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:42:39.235 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591260153.347537_146.1 target=1_1591260081.388568_132.1 2020-06-04T08:42:44.236 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:42:44.237 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:42:44.238 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:42:44.281 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:42:44.282 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:42:44.508 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260081.388568_132.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:41:21.388568Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:42:44.509 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591260153.347537_146.1 target=1_1591260081.388568_132.1 2020-06-04T08:42:49.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:42:49.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:42:49.513 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:42:49.555 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:42:49.556 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:42:49.802 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260081.388568_132.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:41:21.388568Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:42:49.803 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591260153.347537_146.1 target=1_1591260081.388568_132.1 2020-06-04T08:42:54.804 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:42:54.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:42:54.806 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:42:54.847 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:42:54.848 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:42:55.087 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260081.388568_132.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:41:21.388568Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:42:55.088 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591260153.347537_146.1 target=1_1591260081.388568_132.1 2020-06-04T08:43:00.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:43:00.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:43:00.093 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:43:00.138 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:43:00.138 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:43:00.354 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260081.388568_132.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:41:21.388568Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:43:00.354 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591260153.347537_146.1 target=1_1591260081.388568_132.1 2020-06-04T08:43:05.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:43:05.358 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:43:05.360 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:43:05.413 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:43:05.417 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:43:05.639 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260081.388568_132.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:41:21.388568Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:43:05.640 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591260153.347537_146.1 target=1_1591260081.388568_132.1 2020-06-04T08:43:10.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:43:10.643 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:43:10.644 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:43:10.687 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:43:10.688 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:43:10.933 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260081.388568_132.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:41:21.388568Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:43:10.934 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591260153.347537_146.1 target=1_1591260081.388568_132.1 2020-06-04T08:43:15.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:43:15.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:43:15.937 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:43:15.979 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:43:15.980 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:43:16.180 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260081.388568_132.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:41:21.388568Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:43:16.181 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591260153.347537_146.1 target=1_1591260081.388568_132.1 2020-06-04T08:43:21.182 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:43:21.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:43:21.184 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:43:21.226 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:43:21.226 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:43:21.460 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260081.388568_132.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:41:21.388568Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:43:21.461 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591260153.347537_146.1 target=1_1591260081.388568_132.1 2020-06-04T08:43:26.462 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:43:26.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:43:26.463 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:43:26.504 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:43:26.505 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:43:26.737 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260081.388568_132.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:41:21.388568Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:43:26.738 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591260153.347537_146.1 target=1_1591260081.388568_132.1 2020-06-04T08:43:31.739 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:43:31.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:43:31.741 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:43:31.782 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:43:31.783 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:43:32.028 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260081.388568_132.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:41:21.388568Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:43:32.029 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1591260153.347537_146.1 target=1_1591260081.388568_132.1 2020-06-04T08:43:33.460 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:43:33.454+0000 7f53beff5700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:43:33.461 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-06-04T08:43:33.444+0000 7f143f7ee700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:43:37.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:43:37.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:43:37.032 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:43:37.074 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:43:37.074 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:43:37.312 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260153.347537_146.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:42:33.347537Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:43:37.313 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:43:37.314 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:43:37.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:43:37.315 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:43:37.315 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:43:37.359 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:43:37.359 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:43:37.603 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260153.347537_146.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:42:33.347537Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:43:37.604 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:43:37.605 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:43:37.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:43:37.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:43:37.606 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:43:37.648 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:43:37.649 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:43:37.920 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260153.347537_146.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:42:33.347537Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:43:37.920 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:43:37.942 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=7ad4cc51-2d19-48f1-a21e-ef15faf62b94', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:43:37.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=7ad4cc51-2d19-48f1-a21e-ef15faf62b94', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:43:37.944 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=7ad4cc51-2d19-48f1-a21e-ef15faf62b94 --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:43:37.999 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:43:38.000 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:43:38.024 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:43:38.022+0000 7fc2458870c0 0 failed reading obj info from .rgw.root:zone_info.463a7035-f6f9-4bfc-a894-c421122fd949: (2) No such file or directory 2020-06-04T08:43:38.024 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:43:38.022+0000 7fc2458870c0 0 WARNING: could not read zone params for zone id=463a7035-f6f9-4bfc-a894-c421122fd949 name=test-zone3 2020-06-04T08:43:38.026 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:43:38.022+0000 7fc2458870c0 0 failed reading obj info from .rgw.root:zone_info.7ad4cc51-2d19-48f1-a21e-ef15faf62b94: (2) No such file or directory 2020-06-04T08:43:38.026 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:43:38.022+0000 7fc2458870c0 0 WARNING: could not read zone params for zone id=7ad4cc51-2d19-48f1-a21e-ef15faf62b94 name=test-zone1 2020-06-04T08:43:38.038 INFO:teuthology.orchestra.run.smithi196.stdout:{"id":"d7df857b-1e19-4b4c-aa22-43c42d072194","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"M2KEYTP439P9KXXD","secret_key":"Ad56bMSsbqPkjfMTVush7dBgYdLf27ul"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","notif_pool":"test-zone2.rgw.log:notif"} 2020-06-04T08:43:38.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:43:38.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:43:38.040 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:43:38.114 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:43:38.114 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:43:38.332 INFO:teuthology.orchestra.run.smithi196.stderr:Sending period to new master zone 7ad4cc51-2d19-48f1-a21e-ef15faf62b94 2020-06-04T08:43:38.435 INFO:teuthology.orchestra.run.smithi196.stdout:{"id":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","epoch":3,"predecessor_uuid":"8681f463-3379-4ae0-aef4-6adb4b0f951f","sync_status":["1_1591260003.794286_107.1","","",""],"period_map":{"id":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","zones":[{"id":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","name":"test-zone4","endpoints":["http://smithi196.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"463a7035-f6f9-4bfc-a894-c421122fd949","name":"test-zone3","endpoints":["http://smithi060.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","name":"test-zone1","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7df857b-1e19-4b4c-aa22-43c42d072194","name":"test-zone2","endpoints":["http://smithi196.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94"}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","val":2528435237},{"key":"463a7035-f6f9-4bfc-a894-c421122fd949","val":2270195151},{"key":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","val":3611466229},{"key":"d7df857b-1e19-4b4c-aa22-43c42d072194","val":443345077}]},"master_zonegroup":"test-zonegroup","master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","realm_name":"test-realm","realm_epoch":4} 2020-06-04T08:43:38.436 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "7ad4cc51-2d19-48f1-a21e-ef15faf62b94" 2020-06-04T08:44:38.806 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:44:38.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:44:38.809 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:44:38.851 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:44:38.852 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:44:38.878 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:44:38.877+0000 7f8a84e6f0c0 0 failed reading obj info from .rgw.root:zone_info.463a7035-f6f9-4bfc-a894-c421122fd949: (2) No such file or directory 2020-06-04T08:44:38.879 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:44:38.877+0000 7f8a84e6f0c0 0 WARNING: could not read zone params for zone id=463a7035-f6f9-4bfc-a894-c421122fd949 name=test-zone3 2020-06-04T08:44:38.879 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:44:38.877+0000 7f8a84e6f0c0 0 failed reading obj info from .rgw.root:zone_info.7ad4cc51-2d19-48f1-a21e-ef15faf62b94: (2) No such file or directory 2020-06-04T08:44:38.879 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:44:38.877+0000 7f8a84e6f0c0 0 WARNING: could not read zone params for zone id=7ad4cc51-2d19-48f1-a21e-ef15faf62b94 name=test-zone1 2020-06-04T08:44:38.891 INFO:teuthology.orchestra.run.smithi196.stdout:{"id":"d7df857b-1e19-4b4c-aa22-43c42d072194","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"M2KEYTP439P9KXXD","secret_key":"Ad56bMSsbqPkjfMTVush7dBgYdLf27ul"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","notif_pool":"test-zone2.rgw.log:notif"} 2020-06-04T08:44:38.892 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:44:38.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:44:38.893 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:44:38.973 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:44:38.974 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:44:39.192 INFO:teuthology.orchestra.run.smithi196.stderr:Sending period to new master zone 7ad4cc51-2d19-48f1-a21e-ef15faf62b94 2020-06-04T08:44:39.292 INFO:teuthology.orchestra.run.smithi196.stdout:{"id":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","epoch":4,"predecessor_uuid":"8681f463-3379-4ae0-aef4-6adb4b0f951f","sync_status":["1_1591260003.794286_107.1","","",""],"period_map":{"id":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","zones":[{"id":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","name":"test-zone4","endpoints":["http://smithi196.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"463a7035-f6f9-4bfc-a894-c421122fd949","name":"test-zone3","endpoints":["http://smithi060.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","name":"test-zone1","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7df857b-1e19-4b4c-aa22-43c42d072194","name":"test-zone2","endpoints":["http://smithi196.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94"}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","val":2528435237},{"key":"463a7035-f6f9-4bfc-a894-c421122fd949","val":2270195151},{"key":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","val":3611466229},{"key":"d7df857b-1e19-4b4c-aa22-43c42d072194","val":443345077}]},"master_zonegroup":"test-zonegroup","master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","realm_name":"test-realm","realm_epoch":4} 2020-06-04T08:44:39.293 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2020-06-04T08:45:39.295 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:45:39.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:45:39.301 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:45:39.343 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:45:39.344 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:45:39.374 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:45:39.367+0000 7ff46bd950c0 0 failed reading obj info from .rgw.root:zone_info.463a7035-f6f9-4bfc-a894-c421122fd949: (2) No such file or directory 2020-06-04T08:45:39.374 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:45:39.367+0000 7ff46bd950c0 0 WARNING: could not read zone params for zone id=463a7035-f6f9-4bfc-a894-c421122fd949 name=test-zone3 2020-06-04T08:45:39.375 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:45:39.367+0000 7ff46bd950c0 0 failed reading obj info from .rgw.root:zone_info.7ad4cc51-2d19-48f1-a21e-ef15faf62b94: (2) No such file or directory 2020-06-04T08:45:39.375 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:45:39.367+0000 7ff46bd950c0 0 WARNING: could not read zone params for zone id=7ad4cc51-2d19-48f1-a21e-ef15faf62b94 name=test-zone1 2020-06-04T08:45:39.388 INFO:teuthology.orchestra.run.smithi196.stdout:{"id":"d7df857b-1e19-4b4c-aa22-43c42d072194","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"M2KEYTP439P9KXXD","secret_key":"Ad56bMSsbqPkjfMTVush7dBgYdLf27ul"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","notif_pool":"test-zone2.rgw.log:notif"} 2020-06-04T08:45:39.389 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:45:39.390 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:45:39.391 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:45:39.468 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:45:39.468 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:45:39.700 INFO:teuthology.orchestra.run.smithi196.stderr:Sending period to new master zone 7ad4cc51-2d19-48f1-a21e-ef15faf62b94 2020-06-04T08:45:39.793 INFO:teuthology.orchestra.run.smithi196.stdout:{"id":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","epoch":5,"predecessor_uuid":"8681f463-3379-4ae0-aef4-6adb4b0f951f","sync_status":["1_1591260003.794286_107.1","","",""],"period_map":{"id":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","zones":[{"id":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","name":"test-zone4","endpoints":["http://smithi196.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"463a7035-f6f9-4bfc-a894-c421122fd949","name":"test-zone3","endpoints":["http://smithi060.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","name":"test-zone1","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7df857b-1e19-4b4c-aa22-43c42d072194","name":"test-zone2","endpoints":["http://smithi196.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","val":2528435237},{"key":"463a7035-f6f9-4bfc-a894-c421122fd949","val":2270195151},{"key":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","val":3611466229},{"key":"d7df857b-1e19-4b4c-aa22-43c42d072194","val":443345077}]},"master_zonegroup":"test-zonegroup","master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","realm_name":"test-realm","realm_epoch":4} 2020-06-04T08:45:39.794 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2020-06-04T08:46:39.796 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_zone_redirect ... ok 2020-06-04T08:46:39.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:46:39.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:46:39.801 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:46:39.839 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:39.840 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:40.058 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=387a2dc2-99d4-41c4-ad70-50eff9e4eab1 2020-06-04T08:46:40.092 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260153.347537_146.1","last_update":"2020-06-04T08:42:33.347537Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:46:40.093 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:46:40.094 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:46:40.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:46:40.097 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:46:40.138 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:40.139 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:40.368 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260153.347537_146.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:42:33.347537Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:46:40.369 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:46:40.370 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:46:40.371 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:46:40.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:46:40.373 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:46:40.415 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:40.416 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:40.664 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260153.347537_146.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:42:33.347537Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:46:40.664 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:46:40.665 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:46:40.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:46:40.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:46:40.667 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:46:40.709 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:40.710 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:40.983 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260153.347537_146.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:42:33.347537Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:46:40.984 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:46:40.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'M2KEYTP439P9KXXD', '--secret', 'Ad56bMSsbqPkjfMTVush7dBgYdLf27ul', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:40.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'M2KEYTP439P9KXXD', '--secret', 'Ad56bMSsbqPkjfMTVush7dBgYdLf27ul', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:40.988 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key M2KEYTP439P9KXXD --secret Ad56bMSsbqPkjfMTVush7dBgYdLf27ul --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:46:41.070 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:41.071 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:41.128 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:46:41.122+0000 7f9bff82f0c0 0 failed reading obj info from .rgw.root:zone_info.463a7035-f6f9-4bfc-a894-c421122fd949: (2) No such file or directory 2020-06-04T08:46:41.129 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:46:41.122+0000 7f9bff82f0c0 0 WARNING: could not read zone params for zone id=463a7035-f6f9-4bfc-a894-c421122fd949 name=test-zone3 2020-06-04T08:46:41.129 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:46:41.122+0000 7f9bff82f0c0 0 failed reading obj info from .rgw.root:zone_info.7ad4cc51-2d19-48f1-a21e-ef15faf62b94: (2) No such file or directory 2020-06-04T08:46:41.130 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:46:41.122+0000 7f9bff82f0c0 0 WARNING: could not read zone params for zone id=7ad4cc51-2d19-48f1-a21e-ef15faf62b94 name=test-zone1 2020-06-04T08:46:41.137 INFO:teuthology.orchestra.run.smithi196.stdout:{"id":"30b5aa4d-d495-4a5b-af1e-82827c114758","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"M2KEYTP439P9KXXD","secret_key":"Ad56bMSsbqPkjfMTVush7dBgYdLf27ul"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","notif_pool":"remove.rgw.log:notif"} 2020-06-04T08:46:41.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:41.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:41.139 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:46:41.217 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:41.218 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:49.043 INFO:teuthology.orchestra.run.smithi196.stderr:Sending period to new master zone 7ad4cc51-2d19-48f1-a21e-ef15faf62b94 2020-06-04T08:46:49.136 INFO:teuthology.orchestra.run.smithi196.stdout:{"id":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","epoch":6,"predecessor_uuid":"8681f463-3379-4ae0-aef4-6adb4b0f951f","sync_status":["1_1591260003.794286_107.1","","",""],"period_map":{"id":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","zones":[{"id":"30b5aa4d-d495-4a5b-af1e-82827c114758","name":"remove","endpoints":[],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","name":"test-zone4","endpoints":["http://smithi196.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"463a7035-f6f9-4bfc-a894-c421122fd949","name":"test-zone3","endpoints":["http://smithi060.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","name":"test-zone1","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7df857b-1e19-4b4c-aa22-43c42d072194","name":"test-zone2","endpoints":["http://smithi196.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"30b5aa4d-d495-4a5b-af1e-82827c114758","val":3934427836},{"key":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","val":2528435237},{"key":"463a7035-f6f9-4bfc-a894-c421122fd949","val":2270195151},{"key":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","val":3611466229},{"key":"d7df857b-1e19-4b4c-aa22-43c42d072194","val":443345077}]},"master_zonegroup":"test-zonegroup","master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","realm_name":"test-realm","realm_epoch":4} 2020-06-04T08:46:49.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:49.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:49.138 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:46:49.178 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:49.178 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:49.201 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:46:49.191+0000 7fee737980c0 0 failed reading obj info from .rgw.root:zone_info.38aeb3a1-6a1a-479f-a31a-ae49981e3118: (2) No such file or directory 2020-06-04T08:46:49.201 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:46:49.191+0000 7fee737980c0 0 WARNING: could not read zone params for zone id=38aeb3a1-6a1a-479f-a31a-ae49981e3118 name=test-zone4 2020-06-04T08:46:49.203 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:46:49.195+0000 7fee737980c0 0 failed reading obj info from .rgw.root:zone_info.d7df857b-1e19-4b4c-aa22-43c42d072194: (2) No such file or directory 2020-06-04T08:46:49.203 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:46:49.195+0000 7fee737980c0 0 WARNING: could not read zone params for zone id=d7df857b-1e19-4b4c-aa22-43c42d072194 name=test-zone2 2020-06-04T08:46:49.212 INFO:teuthology.orchestra.run.smithi060.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","zones":[{"id":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","name":"test-zone4","endpoints":["http://smithi196.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"463a7035-f6f9-4bfc-a894-c421122fd949","name":"test-zone3","endpoints":["http://smithi060.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","name":"test-zone1","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7df857b-1e19-4b4c-aa22-43c42d072194","name":"test-zone2","endpoints":["http://smithi196.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","sync_policy":{"groups":[]}} 2020-06-04T08:46:49.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:49.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:49.214 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:46:49.297 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:49.298 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:49.317 INFO:teuthology.orchestra.run.smithi060.stderr:zone name remove not found in zonegroup test-zonegroup 2020-06-04T08:46:49.324 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-06-04T08:46:49.325 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:49.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:49.326 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2020-06-04T08:46:49.363 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:49.363 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:49.406 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:46:49.402+0000 7f737ed590c0 0 failed reading obj info from .rgw.root:zone_info.463a7035-f6f9-4bfc-a894-c421122fd949: (2) No such file or directory 2020-06-04T08:46:49.406 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:46:49.402+0000 7f737ed590c0 0 WARNING: could not read zone params for zone id=463a7035-f6f9-4bfc-a894-c421122fd949 name=test-zone3 2020-06-04T08:46:49.406 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:46:49.402+0000 7f737ed590c0 0 failed reading obj info from .rgw.root:zone_info.7ad4cc51-2d19-48f1-a21e-ef15faf62b94: (2) No such file or directory 2020-06-04T08:46:49.407 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:46:49.402+0000 7f737ed590c0 0 WARNING: could not read zone params for zone id=7ad4cc51-2d19-48f1-a21e-ef15faf62b94 name=test-zone1 2020-06-04T08:46:49.410 INFO:teuthology.orchestra.run.smithi196.stderr:2020-06-04T08:46:49.406+0000 7f737ed590c0 0 zone id 30b5aa4d-d495-4a5b-af1e-82827c114758 is not a part of zonegroup default 2020-06-04T08:46:49.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:49.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:49.421 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:46:49.467 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:49.467 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:49.546 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:46:49.539+0000 7f8c79a4a0c0 0 WARNING: can't generate connection for zone 30b5aa4d-d495-4a5b-af1e-82827c114758 id remove: no endpoints defined 2020-06-04T08:46:49.786 INFO:teuthology.orchestra.run.smithi060.stdout:{"id":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","epoch":7,"predecessor_uuid":"8681f463-3379-4ae0-aef4-6adb4b0f951f","sync_status":["1_1591260003.794286_107.1","","",""],"period_map":{"id":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","zones":[{"id":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","name":"test-zone4","endpoints":["http://smithi196.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"463a7035-f6f9-4bfc-a894-c421122fd949","name":"test-zone3","endpoints":["http://smithi060.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","name":"test-zone1","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7df857b-1e19-4b4c-aa22-43c42d072194","name":"test-zone2","endpoints":["http://smithi196.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","val":2528435237},{"key":"463a7035-f6f9-4bfc-a894-c421122fd949","val":2270195151},{"key":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","val":3611466229},{"key":"d7df857b-1e19-4b4c-aa22-43c42d072194","val":443345077}]},"master_zonegroup":"test-zonegroup","master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","realm_name":"test-realm","realm_epoch":4} 2020-06-04T08:46:49.787 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zonegroup_remove ... ok 2020-06-04T08:46:49.788 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:49.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:49.789 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:46:49.827 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:49.828 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:49.861 INFO:teuthology.orchestra.run.smithi060.stdout:{"id":"907d83cf-f29e-41d2-bea9-2e6c9139ca3f","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","sync_policy":{"groups":[]}} 2020-06-04T08:46:49.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:49.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:49.863 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:46:49.947 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:49.947 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:50.007 INFO:teuthology.orchestra.run.smithi060.stdout:{"id":"e0cb8971-a0ce-490c-a7ad-a2757bb896ac","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","notif_pool":"remove.rgw.log:notif"} 2020-06-04T08:46:50.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:50.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:50.008 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:46:50.087 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:50.087 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:50.335 INFO:teuthology.orchestra.run.smithi060.stdout:{"id":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","epoch":8,"predecessor_uuid":"8681f463-3379-4ae0-aef4-6adb4b0f951f","sync_status":["1_1591260003.794286_107.1","","",""],"period_map":{"id":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","zonegroups":[{"id":"907d83cf-f29e-41d2-bea9-2e6c9139ca3f","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"e0cb8971-a0ce-490c-a7ad-a2757bb896ac","zones":[{"id":"e0cb8971-a0ce-490c-a7ad-a2757bb896ac","name":"remove","endpoints":[],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","sync_policy":{"groups":[]}},{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","zones":[{"id":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","name":"test-zone4","endpoints":["http://smithi196.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"463a7035-f6f9-4bfc-a894-c421122fd949","name":"test-zone3","endpoints":["http://smithi060.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","name":"test-zone1","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7df857b-1e19-4b4c-aa22-43c42d072194","name":"test-zone2","endpoints":["http://smithi196.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","val":2528435237},{"key":"463a7035-f6f9-4bfc-a894-c421122fd949","val":2270195151},{"key":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","val":3611466229},{"key":"d7df857b-1e19-4b4c-aa22-43c42d072194","val":443345077},{"key":"e0cb8971-a0ce-490c-a7ad-a2757bb896ac","val":1418255552}]},"master_zonegroup":"test-zonegroup","master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","realm_name":"test-realm","realm_epoch":4} 2020-06-04T08:46:50.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:50.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:50.337 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:46:50.423 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:50.424 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:50.483 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:46:50.475+0000 7f94386e30c0 0 zone id e0cb8971-a0ce-490c-a7ad-a2757bb896ac is not a part of zonegroup test-zonegroup 2020-06-04T08:46:50.484 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:46:50.479+0000 7f94386e30c0 0 zone id e0cb8971-a0ce-490c-a7ad-a2757bb896ac is not a part of zonegroup default 2020-06-04T08:46:50.499 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:50.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:50.501 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:46:50.545 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:50.545 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:50.601 INFO:teuthology.orchestra.run.smithi060.stderr:2020-06-04T08:46:50.595+0000 7f36cac540c0 0 ERROR: zonegroup remove_zg has a non existent master zone 2020-06-04T08:46:50.611 DEBUG:teuthology.orchestra.run:got remote process result: 22 2020-06-04T08:46:50.611 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:50.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:50.612 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:46:50.650 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:50.650 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:50.680 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:50.681 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:50.681 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:46:50.762 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:50.762 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:51.007 INFO:teuthology.orchestra.run.smithi060.stdout:{"id":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","epoch":9,"predecessor_uuid":"8681f463-3379-4ae0-aef4-6adb4b0f951f","sync_status":["1_1591260003.794286_107.1","","",""],"period_map":{"id":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","zones":[{"id":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","name":"test-zone4","endpoints":["http://smithi196.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"463a7035-f6f9-4bfc-a894-c421122fd949","name":"test-zone3","endpoints":["http://smithi060.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","name":"test-zone1","endpoints":["http://smithi060.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"d7df857b-1e19-4b4c-aa22-43c42d072194","name":"test-zone2","endpoints":["http://smithi196.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"38aeb3a1-6a1a-479f-a31a-ae49981e3118","val":2528435237},{"key":"463a7035-f6f9-4bfc-a894-c421122fd949","val":2270195151},{"key":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","val":3611466229},{"key":"d7df857b-1e19-4b4c-aa22-43c42d072194","val":443345077}]},"master_zonegroup":"test-zonegroup","master_zone":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"9c559937-87e1-4867-81f3-e687b7a20f8b","realm_name":"test-realm","realm_epoch":4} 2020-06-04T08:46:51.008 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zg_master_zone_delete ... ok 2020-06-04T08:46:51.009 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-50 2020-06-04T08:46:51.403 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gpxthc-51 2020-06-04T08:46:51.479 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gpxthc-52 2020-06-04T08:46:51.548 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-53 2020-06-04T08:46:51.589 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gpxthc-54 2020-06-04T08:46:51.654 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gpxthc-55 2020-06-04T08:46:53.914 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_set_bucket_website ... tasks.rgw_multi.tests.test_set_bucket_policy ... ok 2020-06-04T08:46:53.916 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-56 2020-06-04T08:46:53.958 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gpxthc-57 2020-06-04T08:46:54.026 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gpxthc-58 2020-06-04T08:46:54.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'gpxthc-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:54.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'gpxthc-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:54.088 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket gpxthc-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:46:54.125 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:54.125 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:54.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'gpxthc-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:54.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'gpxthc-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:54.452 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket gpxthc-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:46:54.534 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:54.534 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:54.852 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'gpxthc-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:54.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'gpxthc-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:46:54.855 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket gpxthc-58 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:46:54.936 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:54.936 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:55.268 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:46:55.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:46:55.269 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:46:55.346 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:55.347 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:55.601 INFO:teuthology.orchestra.run.smithi060.stdout:["obj_delete_at_hint.0000000078","meta.history","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.50-gpxthc-41","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:3","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29-gpxthc-20","datalog.sync-status.shard.463a7035-f6f9-4bfc-a894-c421122fd949.2","obj_delete_at_hint.0000000026","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26-gpxthc-17","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:3","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-36:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.45:10","obj_delete_at_hint.0000000028","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.4-gpxthc-4","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:8","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-15:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.24:1","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28:5","datalog.sync-status.shard.463a7035-f6f9-4bfc-a894-c421122fd949.1","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7501.1-gpxthc-49","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33:8","data_log.3","datalog.sync-status.shard.463a7035-f6f9-4bfc-a894-c421122fd949.0","data.full-sync.index.d7df857b-1e19-4b4c-aa22-43c42d072194.2","data.full-sync.index.d7df857b-1e19-4b4c-aa22-43c42d072194.3","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:1","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.18-gpxthc-9","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35:8","obj_delete_at_hint.0000000076","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33-gpxthc-24","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20-gpxthc-11","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-37:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.46:10","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.5-gpxthc-5","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.16-gpxthc-7","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","datalog.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:8","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27:3","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27:5","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:3","bilog.trim","obj_delete_at_hint.0000000016","data.full-sync.index.463a7035-f6f9-4bfc-a894-c421122fd949.2","obj_delete_at_hint.0000000041","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35-gpxthc-26","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.6-gpxthc-55","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-46:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.3:8","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-13:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.22:1","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.9-gpxthc-58","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27-gpxthc-18","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29:8","obj_delete_at_hint.0000000014","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27:3","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-48:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.5:8","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:8","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:1","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-46:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.3:8","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.5-gpxthc-48","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.40-gpxthc-31","obj_delete_at_hint.0000000121","data_log.1","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34-gpxthc-25","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25-gpxthc-16","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27:1","datalog.sync-status.shard.d7df857b-1e19-4b4c-aa22-43c42d072194.2","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26:5","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31:8","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26:1","obj_delete_at_hint.0000000053","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26:1","obj_delete_at_hint.0000000091","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.22-gpxthc-13","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28:5","obj_delete_at_hint.0000000065","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25:3","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.47-gpxthc-38","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27:5","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34:8","datalog.sync-status.shard.d7df857b-1e19-4b4c-aa22-43c42d072194.3","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.1-gpxthc-44","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-37:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.46:10","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.7-gpxthc-56","mdlog.sync-status.shard.2","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.8-gpxthc-57","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.1-gpxthc-1","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35:8","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-14:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.23:1","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32:8","obj_delete_at_hint.0000000116","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:3","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.4-gpxthc-47","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","data.full-sync.index.463a7035-f6f9-4bfc-a894-c421122fd949.3","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28:8","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31-gpxthc-22","mdlog.sync-status.shard.1","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26:5","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","data.full-sync.index.d7df857b-1e19-4b4c-aa22-43c42d072194.0","obj_delete_at_hint.0000000066","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.5-gpxthc-54","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:1","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-47:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.4:8","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.3-gpxthc-46","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26:3","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:1","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","meta.full-sync.index.3","obj_delete_at_hint.0000000118","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-38:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.47:10","datalog.sync-status.shard.d7df857b-1e19-4b4c-aa22-43c42d072194.0","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.3-gpxthc-3","obj_delete_at_hint.0000000036","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:3","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","mdlog.sync-status.shard.0","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29:8","obj_delete_at_hint.0000000027","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32:8","mdlog.sync-status","obj_delete_at_hint.0000000019","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.23-gpxthc-14","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.46-gpxthc-37","obj_delete_at_hint.0000000039","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.24-gpxthc-15","obj_delete_at_hint.0000000100","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.45-gpxthc-36","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28:8","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25:1","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.4-gpxthc-53","obj_delete_at_hint.0000000093","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31:8","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-48:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.5:8","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.44-gpxthc-35","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.38-gpxthc-29","obj_delete_at_hint.0000000057","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:7","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","data_log.0","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-38:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.47:10","obj_delete_at_hint.0000000012","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:7","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.6-gpxthc-6","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:1","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-49:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7501.1:5","data.full-sync.index.463a7035-f6f9-4bfc-a894-c421122fd949.1","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28-gpxthc-19","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.1-gpxthc-50","obj_delete_at_hint.0000000020","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.39-gpxthc-30","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:7","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:8","obj_delete_at_hint.0000000072","data.full-sync.index.d7df857b-1e19-4b4c-aa22-43c42d072194.1","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-13:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.22:1","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.37-gpxthc-28","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.2-gpxthc-2","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.41-gpxthc-32","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.2-gpxthc-45","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26:3","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32-gpxthc-23","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-15:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.24:1","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25:3","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.42-gpxthc-33","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19-gpxthc-10","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30:8","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34:8","obj_delete_at_hint.0000000013","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-47:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.4:8","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.36-gpxthc-27","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25:5","obj_delete_at_hint.0000000051","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25:1","obj_delete_at_hint.0000000106","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.43-gpxthc-34","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.17-gpxthc-8","obj_delete_at_hint.0000000017","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.49-gpxthc-40","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-49:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7501.1:5","datalog.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949","obj_delete_at_hint.0000000023","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25:5","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:7","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","datalog.sync-status.shard.463a7035-f6f9-4bfc-a894-c421122fd949.3","2020-06-04-08-d7df857b-1e19-4b4c-aa22-43c42d072194.6610.1-gpxthc-42","meta.full-sync.index.2","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33:8","data_log.2","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-36:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.45:10","obj_delete_at_hint.0000000067","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:8","obj_delete_at_hint.0000000086","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:7","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:1","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:7","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.2-gpxthc-51","2020-06-04-08-d7df857b-1e19-4b4c-aa22-43c42d072194.6610.2-gpxthc-43","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30:8","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","datalog.sync-status.shard.d7df857b-1e19-4b4c-aa22-43c42d072194.1","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.48-gpxthc-39","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27:1","meta.log.387a2dc2-99d4-41c4-ad70-50eff9e4eab1.0","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:3","obj_delete_at_hint.0000000124","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.3-gpxthc-52","obj_delete_at_hint.0000000042","data.full-sync.index.463a7035-f6f9-4bfc-a894-c421122fd949.0","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-14:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.23:1","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30-gpxthc-21","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:8","obj_delete_at_hint.0000000075","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21-gpxthc-12","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030"] 2020-06-04T08:46:55.634 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:46:55.635 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:46:55.635 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:46:55.674 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:55.675 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:55.918 INFO:teuthology.orchestra.run.smithi196.stdout:["obj_delete_at_hint.0000000078","meta.history","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.50-gpxthc-41","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-37:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.46:10","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29-gpxthc-20","datalog.sync-status.shard.463a7035-f6f9-4bfc-a894-c421122fd949.2","obj_delete_at_hint.0000000026","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26-gpxthc-17","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:3","obj_delete_at_hint.0000000028","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.4-gpxthc-4","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-13:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.22:1","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30:8","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:7","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25:1","datalog.sync-status.shard.463a7035-f6f9-4bfc-a894-c421122fd949.1","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7501.1-gpxthc-49","data_log.3","datalog.sync-status.shard.463a7035-f6f9-4bfc-a894-c421122fd949.0","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.18-gpxthc-9","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35:8","obj_delete_at_hint.0000000076","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33-gpxthc-24","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20-gpxthc-11","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-37:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.46:10","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.5-gpxthc-5","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.16-gpxthc-7","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","obj_delete_at_hint.0000000096","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:3","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:8","datalog.sync-status.shard.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.0","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27:5","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:3","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:7","bilog.trim","obj_delete_at_hint.0000000016","data.full-sync.index.463a7035-f6f9-4bfc-a894-c421122fd949.2","data.full-sync.index.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.1","obj_delete_at_hint.0000000041","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35-gpxthc-26","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-46:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.3:8","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27:5","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:8","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27-gpxthc-18","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25:3","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33:8","obj_delete_at_hint.0000000014","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27:3","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:8","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:1","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34:8","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.5-gpxthc-48","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.40-gpxthc-31","obj_delete_at_hint.0000000121","data_log.1","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28:5","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29:8","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34-gpxthc-25","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:8","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25-gpxthc-16","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26:5","datalog.sync-status.shard.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.1","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26:1","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.22-gpxthc-13","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28:5","obj_delete_at_hint.0000000065","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25:3","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.47-gpxthc-38","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-46:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.3:8","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34:8","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.1-gpxthc-44","mdlog.sync-status.shard.2","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.8-gpxthc-57","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.1-gpxthc-1","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28:8","obj_delete_at_hint.0000000116","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.4-gpxthc-47","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","data.full-sync.index.463a7035-f6f9-4bfc-a894-c421122fd949.3","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28:8","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31-gpxthc-22","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-49:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7501.1:5","mdlog.sync-status.shard.1","obj_delete_at_hint.0000000079","datalog.sync-status.shard.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.3","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","obj_delete_at_hint.0000000066","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.5-gpxthc-54","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:1","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.3-gpxthc-46","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26:3","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27:1","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","datalog.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94","meta.full-sync.index.3","obj_delete_at_hint.0000000118","data.full-sync.index.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.2","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-38:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.47:10","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.3-gpxthc-3","obj_delete_at_hint.0000000036","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:3","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25:5","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","data.full-sync.index.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.0","mdlog.sync-status.shard.0","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29:8","obj_delete_at_hint.0000000027","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32:8","mdlog.sync-status","obj_delete_at_hint.0000000019","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.23-gpxthc-14","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.46-gpxthc-37","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-47:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.4:8","obj_delete_at_hint.0000000039","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.24-gpxthc-15","obj_delete_at_hint.0000000100","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.45-gpxthc-36","datalog.sync-status.shard.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.2","obj_delete_at_hint.0000000093","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31:8","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-48:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.5:8","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27:3","obj_delete_at_hint.0000000057","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:7","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-36:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.45:10","data_log.0","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:8","obj_delete_at_hint.0000000012","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:7","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.6-gpxthc-6","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-48:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.5:8","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:3","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-38:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.47:10","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:7","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-49:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7501.1:5","data.full-sync.index.463a7035-f6f9-4bfc-a894-c421122fd949.1","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28-gpxthc-19","obj_delete_at_hint.0000000020","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:8","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:3","obj_delete_at_hint.0000000072","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-13:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.22:1","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31:8","data.full-sync.index.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.3","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.37-gpxthc-28","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-14:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.23:1","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:1","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.2-gpxthc-2","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.2-gpxthc-45","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32-gpxthc-23","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-15:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.24:1","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26:5","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:1","obj_delete_at_hint.0000000109","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19-gpxthc-10","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30:8","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:1","obj_delete_at_hint.0000000013","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-15:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.24:1","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-47:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.4:8","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25:5","obj_delete_at_hint.0000000051","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25:1","obj_delete_at_hint.0000000106","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.43-gpxthc-34","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.17-gpxthc-8","obj_delete_at_hint.0000000017","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.49-gpxthc-40","datalog.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949","obj_delete_at_hint.0000000023","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","datalog.sync-status.shard.463a7035-f6f9-4bfc-a894-c421122fd949.3","2020-06-04-08-d7df857b-1e19-4b4c-aa22-43c42d072194.6610.1-gpxthc-42","meta.full-sync.index.2","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33:8","data_log.2","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-36:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.45:10","obj_delete_at_hint.0000000067","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26:3","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32:8","obj_delete_at_hint.0000000086","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35:8","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:1","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:7","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.2-gpxthc-51","2020-06-04-08-d7df857b-1e19-4b4c-aa22-43c42d072194.6610.2-gpxthc-43","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.48-gpxthc-39","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27:1","meta.log.387a2dc2-99d4-41c4-ad70-50eff9e4eab1.0","obj_delete_at_hint.0000000124","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","data.full-sync.index.463a7035-f6f9-4bfc-a894-c421122fd949.0","bucket.sync-status.463a7035-f6f9-4bfc-a894-c421122fd949:gpxthc-14:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.23:1","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30-gpxthc-21","obj_delete_at_hint.0000000075","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21-gpxthc-12","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26:1"] 2020-06-04T08:46:55.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:46:55.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:46:55.952 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:46:55.992 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:56.000 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:56.238 INFO:teuthology.orchestra.run.smithi060.stdout:["obj_delete_at_hint.0000000078","meta.history","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.50-gpxthc-41","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-37:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.46:10","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:3","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29-gpxthc-20","obj_delete_at_hint.0000000026","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26-gpxthc-17","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-36:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.45:10","obj_delete_at_hint.0000000028","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.4-gpxthc-4","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-13:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.22:1","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:8","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-15:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.24:1","obj_delete_at_hint.0000000069","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30:8","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:7","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28:5","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25:1","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7501.1-gpxthc-49","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33:8","data_log.3","data.full-sync.index.d7df857b-1e19-4b4c-aa22-43c42d072194.2","data.full-sync.index.d7df857b-1e19-4b4c-aa22-43c42d072194.3","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:1","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.18-gpxthc-9","obj_delete_at_hint.0000000076","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33-gpxthc-24","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20-gpxthc-11","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.5-gpxthc-5","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.16-gpxthc-7","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","datalog.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194","obj_delete_at_hint.0000000096","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:3","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","datalog.sync-status.shard.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.0","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27:3","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:7","bilog.trim","obj_delete_at_hint.0000000016","data.full-sync.index.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.1","obj_delete_at_hint.0000000041","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35-gpxthc-26","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.6-gpxthc-55","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-13:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.22:1","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27:5","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:8","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.9-gpxthc-58","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27-gpxthc-18","meta.log.838ed6b2-fa2f-4917-a5c1-6c82bb405cda.1","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25:3","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29:8","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33:8","obj_delete_at_hint.0000000014","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-48:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.5:8","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34:8","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-46:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.3:8","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.5-gpxthc-48","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","data_log.1","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28:5","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29:8","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34-gpxthc-25","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:8","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25-gpxthc-16","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27:1","datalog.sync-status.shard.d7df857b-1e19-4b4c-aa22-43c42d072194.2","datalog.sync-status.shard.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.1","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31:8","obj_delete_at_hint.0000000053","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26:1","obj_delete_at_hint.0000000091","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.22-gpxthc-13","obj_delete_at_hint.0000000065","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.47-gpxthc-38","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27:5","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-46:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.3:8","datalog.sync-status.shard.d7df857b-1e19-4b4c-aa22-43c42d072194.3","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.1-gpxthc-44","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-37:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.46:10","mdlog.sync-status.shard.2","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.1-gpxthc-1","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35:8","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-14:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.23:1","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32:8","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28:8","obj_delete_at_hint.0000000116","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:3","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.4-gpxthc-47","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31-gpxthc-22","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-49:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7501.1:5","mdlog.sync-status.shard.1","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26:5","obj_delete_at_hint.0000000079","datalog.sync-status.shard.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.3","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","data.full-sync.index.d7df857b-1e19-4b4c-aa22-43c42d072194.0","obj_delete_at_hint.0000000066","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-47:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.4:8","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.3-gpxthc-46","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:1","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27:1","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","datalog.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94","meta.full-sync.index.3","obj_delete_at_hint.0000000118","data.full-sync.index.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.2","datalog.sync-status.shard.d7df857b-1e19-4b4c-aa22-43c42d072194.0","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.3-gpxthc-3","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25:5","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","data.full-sync.index.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.0","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.23-gpxthc-14","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.46-gpxthc-37","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-47:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.4:8","obj_delete_at_hint.0000000039","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.24-gpxthc-15","obj_delete_at_hint.0000000100","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.45-gpxthc-36","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28:8","datalog.sync-status.shard.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.2","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25:1","obj_delete_at_hint.0000000093","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.44-gpxthc-35","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27:3","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.38-gpxthc-29","obj_delete_at_hint.0000000057","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-36:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.45:10","data_log.0","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:8","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-38:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.47:10","obj_delete_at_hint.0000000012","meta.log.8681f463-3379-4ae0-aef4-6adb4b0f951f.0","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.6-gpxthc-6","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-48:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.5:8","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:3","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:1","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-38:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.47:10","meta.log.838ed6b2-fa2f-4917-a5c1-6c82bb405cda.0","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:7","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28-gpxthc-19","obj_delete_at_hint.0000000020","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:7","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:3","obj_delete_at_hint.0000000072","data.full-sync.index.d7df857b-1e19-4b4c-aa22-43c42d072194.1","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31:8","data.full-sync.index.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.3","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-14:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.23:1","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:1","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.2-gpxthc-2","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.41-gpxthc-32","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.2-gpxthc-45","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26:3","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32-gpxthc-23","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25:3","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26:5","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:1","obj_delete_at_hint.0000000109","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19-gpxthc-10","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:1","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34:8","obj_delete_at_hint.0000000013","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-15:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.24:1","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.17-gpxthc-8","obj_delete_at_hint.0000000017","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.49-gpxthc-40","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-49:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7501.1:5","obj_delete_at_hint.0000000023","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25:5","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:7","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","2020-06-04-08-d7df857b-1e19-4b4c-aa22-43c42d072194.6610.1-gpxthc-42","meta.full-sync.index.2","data_log.2","obj_delete_at_hint.0000000067","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:8","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26:3","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32:8","obj_delete_at_hint.0000000086","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:7","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35:8","2020-06-04-08-d7df857b-1e19-4b4c-aa22-43c42d072194.6610.2-gpxthc-43","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30:8","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","datalog.sync-status.shard.d7df857b-1e19-4b4c-aa22-43c42d072194.1","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.48-gpxthc-39","meta.log.387a2dc2-99d4-41c4-ad70-50eff9e4eab1.0","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:3","obj_delete_at_hint.0000000124","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.3-gpxthc-52","obj_delete_at_hint.0000000042","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30-gpxthc-21","bucket.sync-status.d7df857b-1e19-4b4c-aa22-43c42d072194:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:8","obj_delete_at_hint.0000000075","2020-06-04-08-7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21-gpxthc-12","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26:1"] 2020-06-04T08:46:56.270 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:46:56.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:46:56.272 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:46:56.308 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:56.308 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:56.539 INFO:teuthology.orchestra.run.smithi196.stdout:["obj_delete_at_hint.0000000078","meta.history","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-37:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.46:10","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","obj_delete_at_hint.0000000026","obj_delete_at_hint.0000000028","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-13:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.22:1","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-21:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.30:8","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:7","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25:1","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","obj_delete_at_hint.0000000096","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:3","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","datalog.sync-status.shard.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.0","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:7","bilog.trim","obj_delete_at_hint.0000000016","data.full-sync.index.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.1","obj_delete_at_hint.0000000041","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27:5","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:8","meta.log.838ed6b2-fa2f-4917-a5c1-6c82bb405cda.1","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25:3","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-24:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.33:8","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-25:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.34:8","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28:5","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-20:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.29:8","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:8","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","datalog.sync-status.shard.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.1","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-46:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.3:8","mdlog.sync-status.shard.2","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-19:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.28:8","obj_delete_at_hint.0000000116","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-49:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7501.1:5","mdlog.sync-status.shard.1","obj_delete_at_hint.0000000079","datalog.sync-status.shard.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.3","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","obj_delete_at_hint.0000000066","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27:1","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","datalog.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94","meta.full-sync.index.3","obj_delete_at_hint.0000000118","data.full-sync.index.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.2","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-16:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.25:5","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","data.full-sync.index.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.0","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-47:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.4:8","obj_delete_at_hint.0000000039","obj_delete_at_hint.0000000100","datalog.sync-status.shard.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.2","obj_delete_at_hint.0000000093","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-18:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.27:3","obj_delete_at_hint.0000000057","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-36:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.45:10","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:8","obj_delete_at_hint.0000000012","meta.log.8681f463-3379-4ae0-aef4-6adb4b0f951f.0","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-48:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7175.5:8","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:3","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-38:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.47:10","meta.log.838ed6b2-fa2f-4917-a5c1-6c82bb405cda.0","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:7","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:3","obj_delete_at_hint.0000000072","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-22:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.31:8","data.full-sync.index.7ad4cc51-2d19-48f1-a21e-ef15faf62b94.3","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-14:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.23:1","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-11:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.20:1","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26:5","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-12:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.21:1","obj_delete_at_hint.0000000109","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-10:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.19:1","obj_delete_at_hint.0000000013","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-15:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.24:1","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","obj_delete_at_hint.0000000023","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","meta.full-sync.index.2","obj_delete_at_hint.0000000067","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26:3","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-23:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.32:8","obj_delete_at_hint.0000000086","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-26:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.35:8","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","meta.log.387a2dc2-99d4-41c4-ad70-50eff9e4eab1.0","obj_delete_at_hint.0000000124","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","bucket.sync-status.7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-17:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.4470.26:1"] 2020-06-04T08:46:56.587 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:46:56.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:46:56.589 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:46:56.630 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:56.631 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:56.850 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260415.230785_210.3","last_update":"2020-06-04T08:46:55.230785Z"},{"marker":"1_1591260415.230785_459.3","last_update":"2020-06-04T08:46:55.230785Z"},{"marker":"1_1591260415.230785_278.2","last_update":"2020-06-04T08:46:55.230785Z"},{"marker":"1_1591260415.230785_488.3","last_update":"2020-06-04T08:46:55.230785Z"}] 2020-06-04T08:46:56.851 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-06-04T08:46:56.852 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:46:56.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:46:56.853 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:46:56.902 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:46:56.902 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:46:57.126 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11207024744551397021},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.317599_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.317599Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.322360_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.322360Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:46:57.127 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1591260415.230785_210.3 target=1_1591260279.317599_148.1, shard=1 master=1_1591260415.230785_459.3 target=1_1591259930.643002_247.1, shard=2 master=1_1591260415.230785_278.2 target=1_1591259589.882369_86.1, shard=3 master=1_1591260415.230785_488.3 target=1_1591260085.322360_320.1 2020-06-04T08:47:02.128 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:02.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:02.131 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:47:02.175 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:47:02.175 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:47:02.415 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11207024744551397021},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.317599_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.317599Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.322360_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.322360Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:47:02.416 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1591260415.230785_210.3 target=1_1591260279.317599_148.1, shard=1 master=1_1591260415.230785_459.3 target=1_1591259930.643002_247.1, shard=2 master=1_1591260415.230785_278.2 target=1_1591259589.882369_86.1, shard=3 master=1_1591260415.230785_488.3 target=1_1591260085.322360_320.1 2020-06-04T08:47:07.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:07.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:07.419 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:47:07.460 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:47:07.461 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:47:07.695 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11207024744551397021},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.317599_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.317599Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.322360_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.322360Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:47:07.695 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1591260415.230785_210.3 target=1_1591260279.317599_148.1, shard=1 master=1_1591260415.230785_459.3 target=1_1591259930.643002_247.1, shard=2 master=1_1591260415.230785_278.2 target=1_1591259589.882369_86.1, shard=3 master=1_1591260415.230785_488.3 target=1_1591260085.322360_320.1 2020-06-04T08:47:12.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:12.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:12.698 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:47:12.740 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:47:12.741 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:47:12.975 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11207024744551397021},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.317599_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.317599Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.322360_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.322360Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:47:12.976 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1591260415.230785_210.3 target=1_1591260279.317599_148.1, shard=1 master=1_1591260415.230785_459.3 target=1_1591259930.643002_247.1, shard=2 master=1_1591260415.230785_278.2 target=1_1591259589.882369_86.1, shard=3 master=1_1591260415.230785_488.3 target=1_1591260085.322360_320.1 2020-06-04T08:47:17.978 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:17.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:17.980 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:47:18.027 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:47:18.027 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:47:18.250 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11207024744551397021},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.317599_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.317599Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.322360_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.322360Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:47:18.251 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1591260415.230785_210.3 target=1_1591260279.317599_148.1, shard=1 master=1_1591260415.230785_459.3 target=1_1591259930.643002_247.1, shard=2 master=1_1591260415.230785_278.2 target=1_1591259589.882369_86.1, shard=3 master=1_1591260415.230785_488.3 target=1_1591260085.322360_320.1 2020-06-04T08:47:23.253 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:23.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:23.255 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:47:23.297 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:47:23.297 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:47:23.539 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11207024744551397021},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.317599_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.317599Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.322360_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.322360Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:47:23.540 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1591260415.230785_210.3 target=1_1591260279.317599_148.1, shard=1 master=1_1591260415.230785_459.3 target=1_1591259930.643002_247.1, shard=2 master=1_1591260415.230785_278.2 target=1_1591259589.882369_86.1, shard=3 master=1_1591260415.230785_488.3 target=1_1591260085.322360_320.1 2020-06-04T08:47:28.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:28.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:28.543 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:47:28.585 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:47:28.586 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:47:28.827 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11207024744551397021},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.317599_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.317599Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.322360_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.322360Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:47:28.828 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1591260415.230785_210.3 target=1_1591260279.317599_148.1, shard=1 master=1_1591260415.230785_459.3 target=1_1591259930.643002_247.1, shard=2 master=1_1591260415.230785_278.2 target=1_1591259589.882369_86.1, shard=3 master=1_1591260415.230785_488.3 target=1_1591260085.322360_320.1 2020-06-04T08:47:33.830 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:33.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:33.832 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:47:33.873 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:47:33.874 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:47:34.102 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11207024744551397021},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.317599_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.317599Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.322360_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.322360Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:47:34.103 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1591260415.230785_210.3 target=1_1591260279.317599_148.1, shard=1 master=1_1591260415.230785_459.3 target=1_1591259930.643002_247.1, shard=2 master=1_1591260415.230785_278.2 target=1_1591259589.882369_86.1, shard=3 master=1_1591260415.230785_488.3 target=1_1591260085.322360_320.1 2020-06-04T08:47:39.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:39.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:39.106 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:47:39.149 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:47:39.149 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:47:39.396 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11207024744551397021},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.317599_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.317599Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.322360_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.322360Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:47:39.397 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1591260415.230785_210.3 target=1_1591260279.317599_148.1, shard=1 master=1_1591260415.230785_459.3 target=1_1591259930.643002_247.1, shard=2 master=1_1591260415.230785_278.2 target=1_1591259589.882369_86.1, shard=3 master=1_1591260415.230785_488.3 target=1_1591260085.322360_320.1 2020-06-04T08:47:44.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:44.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:44.412 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:47:44.465 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:47:44.466 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:47:44.691 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11207024744551397021},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.317599_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.317599Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.322360_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.322360Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:47:44.691 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1591260415.230785_210.3 target=1_1591260279.317599_148.1, shard=1 master=1_1591260415.230785_459.3 target=1_1591259930.643002_247.1, shard=2 master=1_1591260415.230785_278.2 target=1_1591259589.882369_86.1, shard=3 master=1_1591260415.230785_488.3 target=1_1591260085.322360_320.1 2020-06-04T08:47:49.693 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:49.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:49.694 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:47:49.737 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:47:49.738 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:47:49.982 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11207024744551397021},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.317599_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.317599Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.322360_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.322360Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:47:49.983 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1591260415.230785_210.3 target=1_1591260279.317599_148.1, shard=1 master=1_1591260415.230785_459.3 target=1_1591259930.643002_247.1, shard=2 master=1_1591260415.230785_278.2 target=1_1591259589.882369_86.1, shard=3 master=1_1591260415.230785_488.3 target=1_1591260085.322360_320.1 2020-06-04T08:47:52.419 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:47:52.417+0000 7f84df7f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:47:52.420 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:47:52.417+0000 7f84dfff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:47:52.421 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:47:52.417+0000 7f84dc7f0700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:47:52.422 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:47:52.417+0000 7f84db7ee700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:47:52.422 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:47:52.417+0000 7f85067fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:47:52.427 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:47:52.421+0000 7f84e27fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:47:52.427 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:47:52.421+0000 7f84dafed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:47:52.427 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:47:52.421+0000 7f84dbfef700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:47:52.428 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:47:52.421+0000 7f84ea7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:47:52.428 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:47:52.421+0000 7f84dd7f2700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:47:52.428 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:47:52.421+0000 7f84eb7fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:47:52.429 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:47:52.421+0000 7f84e37fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:47:52.431 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:47:52.421+0000 7f85067fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:47:52.431 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:47:52.421+0000 7f84df7f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:47:52.431 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:47:52.421+0000 7f84eaffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:47:52.431 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:47:52.421+0000 7f84e27fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:47:52.432 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:47:52.425+0000 7f84e3fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:47:52.432 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:47:52.425+0000 7f8504ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:47:52.432 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:47:52.425+0000 7f84de7f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:47:52.433 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:47:52.425+0000 7f84e0ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:47:52.433 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:47:52.425+0000 7f84e8ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:47:52.433 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:47:52.425+0000 7f84e07f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:47:54.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:54.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:54.989 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:47:55.037 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:47:55.037 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:47:55.278 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11207024744551397021},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260415.230785_210.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:46:55.230785Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591260415.230785_459.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:46:55.230785Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591260415.230785_278.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:46:55.230785Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260415.230785_488.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:46:55.230785Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:47:55.279 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-06-04T08:47:55.280 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:55.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:55.281 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:47:55.321 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:47:55.321 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:47:55.572 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260415.230785_210.3","last_update":"2020-06-04T08:46:55.230785Z"},{"marker":"1_1591260415.230785_459.3","last_update":"2020-06-04T08:46:55.230785Z"},{"marker":"1_1591260415.230785_278.2","last_update":"2020-06-04T08:46:55.230785Z"},{"marker":"1_1591260415.230785_488.3","last_update":"2020-06-04T08:46:55.230785Z"}] 2020-06-04T08:47:55.573 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-06-04T08:47:55.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:55.574 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:47:55.574 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:47:55.659 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:47:55.660 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:47:55.891 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2627143167498148538},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.317599_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.317599Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.322360_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.322360Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:47:55.892 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1591260415.230785_210.3 target=1_1591260279.317599_148.1, shard=1 master=1_1591260415.230785_459.3 target=1_1591259930.643002_247.1, shard=2 master=1_1591260415.230785_278.2 target=1_1591259589.882369_86.1, shard=3 master=1_1591260415.230785_488.3 target=1_1591260085.322360_320.1 2020-06-04T08:48:00.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:00.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:00.896 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:00.940 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:00.940 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:01.189 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2627143167498148538},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.317599_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.317599Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.322360_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.322360Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:48:01.190 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1591260415.230785_210.3 target=1_1591260279.317599_148.1, shard=1 master=1_1591260415.230785_459.3 target=1_1591259930.643002_247.1, shard=2 master=1_1591260415.230785_278.2 target=1_1591259589.882369_86.1, shard=3 master=1_1591260415.230785_488.3 target=1_1591260085.322360_320.1 2020-06-04T08:48:06.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:06.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:06.194 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:06.238 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:06.239 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:06.488 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2627143167498148538},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.317599_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.317599Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.322360_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.322360Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:48:06.489 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1591260415.230785_210.3 target=1_1591260279.317599_148.1, shard=1 master=1_1591260415.230785_459.3 target=1_1591259930.643002_247.1, shard=2 master=1_1591260415.230785_278.2 target=1_1591259589.882369_86.1, shard=3 master=1_1591260415.230785_488.3 target=1_1591260085.322360_320.1 2020-06-04T08:48:11.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:11.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:11.491 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:11.532 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:11.533 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:11.766 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2627143167498148538},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.317599_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.317599Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.322360_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.322360Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:48:11.767 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1591260415.230785_210.3 target=1_1591260279.317599_148.1, shard=1 master=1_1591260415.230785_459.3 target=1_1591259930.643002_247.1, shard=2 master=1_1591260415.230785_278.2 target=1_1591259589.882369_86.1, shard=3 master=1_1591260415.230785_488.3 target=1_1591260085.322360_320.1 2020-06-04T08:48:16.768 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:16.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:16.770 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:16.816 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:16.817 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:17.059 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2627143167498148538},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.317599_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.317599Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.322360_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.322360Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:48:17.060 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1591260415.230785_210.3 target=1_1591260279.317599_148.1, shard=1 master=1_1591260415.230785_459.3 target=1_1591259930.643002_247.1, shard=2 master=1_1591260415.230785_278.2 target=1_1591259589.882369_86.1, shard=3 master=1_1591260415.230785_488.3 target=1_1591260085.322360_320.1 2020-06-04T08:48:22.065 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:22.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:22.067 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:22.128 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:22.128 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:22.380 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2627143167498148538},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.317599_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.317599Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.322360_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.322360Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:48:22.381 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1591260415.230785_210.3 target=1_1591260279.317599_148.1, shard=1 master=1_1591260415.230785_459.3 target=1_1591259930.643002_247.1, shard=2 master=1_1591260415.230785_278.2 target=1_1591259589.882369_86.1, shard=3 master=1_1591260415.230785_488.3 target=1_1591260085.322360_320.1 2020-06-04T08:48:27.382 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:27.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:27.384 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:27.430 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:27.430 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:27.673 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2627143167498148538},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.317599_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.317599Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.322360_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.322360Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:48:27.674 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1591260415.230785_210.3 target=1_1591260279.317599_148.1, shard=1 master=1_1591260415.230785_459.3 target=1_1591259930.643002_247.1, shard=2 master=1_1591260415.230785_278.2 target=1_1591259589.882369_86.1, shard=3 master=1_1591260415.230785_488.3 target=1_1591260085.322360_320.1 2020-06-04T08:48:32.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:32.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:32.677 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:32.722 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:32.722 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:32.970 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2627143167498148538},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.317599_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.317599Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.322360_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.322360Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:48:32.970 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1591260415.230785_210.3 target=1_1591260279.317599_148.1, shard=1 master=1_1591260415.230785_459.3 target=1_1591259930.643002_247.1, shard=2 master=1_1591260415.230785_278.2 target=1_1591259589.882369_86.1, shard=3 master=1_1591260415.230785_488.3 target=1_1591260085.322360_320.1 2020-06-04T08:48:37.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:37.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:37.974 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:38.017 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:38.017 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:38.268 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2627143167498148538},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.317599_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.317599Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.322360_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.322360Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:48:38.269 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1591260415.230785_210.3 target=1_1591260279.317599_148.1, shard=1 master=1_1591260415.230785_459.3 target=1_1591259930.643002_247.1, shard=2 master=1_1591260415.230785_278.2 target=1_1591259589.882369_86.1, shard=3 master=1_1591260415.230785_488.3 target=1_1591260085.322360_320.1 2020-06-04T08:48:43.271 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:43.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:43.273 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:43.317 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:43.317 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:43.557 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2627143167498148538},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.317599_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.317599Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.322360_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.322360Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:48:43.557 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1591260415.230785_210.3 target=1_1591260279.317599_148.1, shard=1 master=1_1591260415.230785_459.3 target=1_1591259930.643002_247.1, shard=2 master=1_1591260415.230785_278.2 target=1_1591259589.882369_86.1, shard=3 master=1_1591260415.230785_488.3 target=1_1591260085.322360_320.1 2020-06-04T08:48:48.573 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:48.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:48.576 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:48.621 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:48.622 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:48.860 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2627143167498148538},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.317599_148.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.317599Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.643002_247.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.643002Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.882369_86.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.882369Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.322360_320.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.322360Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:48:48.866 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1591260415.230785_210.3 target=1_1591260279.317599_148.1, shard=1 master=1_1591260415.230785_459.3 target=1_1591259930.643002_247.1, shard=2 master=1_1591260415.230785_278.2 target=1_1591259589.882369_86.1, shard=3 master=1_1591260415.230785_488.3 target=1_1591260085.322360_320.1 2020-06-04T08:48:52.470 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-06-04T08:48:52.462+0000 7f14417f2700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.471 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-06-04T08:48:52.462+0000 7f1441ff3700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.471 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-06-04T08:48:52.462+0000 7f143e7ec700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.471 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-06-04T08:48:52.462+0000 7f145a7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.472 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-06-04T08:48:52.462+0000 7f14467fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.472 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-06-04T08:48:52.462+0000 7f143d7ea700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.538 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.531+0000 7f53d67fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.538 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.531+0000 7f53b7fe7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.539 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.531+0000 7f53c27fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.539 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.531+0000 7f53bcff1700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.539 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.531+0000 7f53c0ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.539 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.531+0000 7f53b97ea700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.539 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.531+0000 7f53c1ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.540 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.535+0000 7f53bd7f2700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.547 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.535+0000 7f53c07f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.547 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.535+0000 7f53bcff1700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.547 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.535+0000 7f53b97ea700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.547 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.535+0000 7f53bb7ee700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.548 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.535+0000 7f53b9feb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.548 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.535+0000 7f53bdff3700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.548 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.535+0000 7f53bc7f0700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.548 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.535+0000 7f53bbfef700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.550 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.535+0000 7f53bafed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.550 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.535+0000 7f53d77fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.550 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.535+0000 7f53d57fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.551 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.535+0000 7f53ba7ec700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.551 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.535+0000 7f53b87e8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.551 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.535+0000 7f53d6ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.551 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.535+0000 7f53b8fe9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.551 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.535+0000 7f53c17fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.552 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.535+0000 7f53c3fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.552 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.535+0000 7f53d5ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.552 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.539+0000 7f53d4ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.552 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.539+0000 7f53b77e6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.552 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.539+0000 7f53c2ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.553 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.539+0000 7f53d67fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.553 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.539+0000 7f53c27fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.553 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.539+0000 7f53b7fe7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:52.553 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:48:52.539+0000 7f53c37fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:48:53.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:53.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:53.870 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:53.913 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:53.914 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:54.158 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2627143167498148538},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260415.230785_210.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:46:55.230785Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591260415.230785_459.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:46:55.230785Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591260415.230785_278.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:46:55.230785Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260415.230785_488.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:46:55.230785Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:48:54.159 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-06-04T08:48:54.159 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:54.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:54.161 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:54.246 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:54.246 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:54.494 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260415.230785_210.3","last_update":"2020-06-04T08:46:55.230785Z"},{"marker":"1_1591260415.230785_459.3","last_update":"2020-06-04T08:46:55.230785Z"},{"marker":"1_1591260415.230785_278.2","last_update":"2020-06-04T08:46:55.230785Z"},{"marker":"1_1591260415.230785_488.3","last_update":"2020-06-04T08:46:55.230785Z"}] 2020-06-04T08:48:54.495 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-06-04T08:48:54.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:54.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:54.496 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:54.541 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:54.541 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:54.820 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7644369918941386260},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260415.230785_210.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:46:55.230785Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591260415.230785_459.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:46:55.230785Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591260415.230785_278.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:46:55.230785Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260415.230785_488.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:46:55.230785Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:48:54.821 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-06-04T08:48:54.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:54.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:54.822 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:54.902 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:54.902 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:55.133 INFO:teuthology.orchestra.run.smithi196.stdout:[{"marker":"1_1591260279.742039_145.1","last_update":"2020-06-04T08:44:39.742039Z"},{"marker":"1_1591259930.639528_257.1","last_update":"2020-06-04T08:38:50.639528Z"},{"marker":"1_1591259589.701718_107.1","last_update":"2020-06-04T08:33:09.701718Z"},{"marker":"1_1591260085.387675_315.1","last_update":"2020-06-04T08:41:25.387675Z"}] 2020-06-04T08:48:55.133 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-06-04T08:48:55.134 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:55.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:55.135 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:55.178 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:55.181 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:55.439 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1308327179334993417},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.742039_145.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.742039Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.639528_257.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.639528Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.701718_107.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.701718Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.387675_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.387675Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:48:55.440 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-06-04T08:48:55.441 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:55.441 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:55.442 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:55.487 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:55.488 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:55.721 INFO:teuthology.orchestra.run.smithi196.stdout:[{"marker":"1_1591260279.742039_145.1","last_update":"2020-06-04T08:44:39.742039Z"},{"marker":"1_1591259930.639528_257.1","last_update":"2020-06-04T08:38:50.639528Z"},{"marker":"1_1591259589.701718_107.1","last_update":"2020-06-04T08:33:09.701718Z"},{"marker":"1_1591260085.387675_315.1","last_update":"2020-06-04T08:41:25.387675Z"}] 2020-06-04T08:48:55.721 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-06-04T08:48:55.722 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:55.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:55.723 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:55.764 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:55.765 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:55.998 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":16507416453976119466},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260279.742039_145.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:44:39.742039Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591259930.639528_257.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:38:50.639528Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591259589.701718_107.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:33:09.701718Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260085.387675_315.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:41:25.387675Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:48:55.999 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-06-04T08:48:55.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:56.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:56.000 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:56.079 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:56.079 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:56.320 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260535.818761_168.1","last_update":"2020-06-04T08:48:55.818761Z"},{"marker":"1_1591260535.823444_365.1","last_update":"2020-06-04T08:48:55.823444Z"},{"marker":"1_1591260535.808145_219.1","last_update":"2020-06-04T08:48:55.808145Z"},{"marker":"1_1591260535.811797_282.1","last_update":"2020-06-04T08:48:55.811797Z"}] 2020-06-04T08:48:56.320 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-06-04T08:48:56.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:56.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:56.322 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:56.407 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:56.407 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:56.660 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3830535810264008500},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260535.818761_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:48:55.818761Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591260535.823444_365.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:48:55.823444Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591260535.808145_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:48:55.808145Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260535.811797_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:48:55.811797Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:48:56.660 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-06-04T08:48:56.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:56.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:56.662 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:56.742 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:56.742 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:56.986 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260535.818761_168.1","last_update":"2020-06-04T08:48:55.818761Z"},{"marker":"1_1591260535.823444_365.1","last_update":"2020-06-04T08:48:55.823444Z"},{"marker":"1_1591260535.808145_219.1","last_update":"2020-06-04T08:48:55.808145Z"},{"marker":"1_1591260535.811797_282.1","last_update":"2020-06-04T08:48:55.811797Z"}] 2020-06-04T08:48:56.986 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-06-04T08:48:56.987 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:56.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:56.988 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:57.029 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:57.030 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:57.268 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9078143951431167771},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260535.818761_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:48:55.818761Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591260535.823444_365.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:48:55.823444Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591260535.808145_219.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:48:55.808145Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260535.811797_282.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:48:55.811797Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:48:57.268 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-06-04T08:48:57.270 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable ... ok 2020-06-04T08:48:57.271 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-59 2020-06-04T08:48:57.315 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gpxthc-60 2020-06-04T08:48:57.389 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gpxthc-61 2020-06-04T08:48:58.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:58.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:58.064 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:58.126 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:58.126 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:58.345 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.56.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.62.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.43.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.37.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:48:58.345 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-59 2020-06-04T08:48:58.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:58.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:58.347 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:58.394 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:58.395 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:58.592 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-06-04T08:48:57.631247Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-04T08:48:57.675393Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5","timestamp":"2020-06-04T08:48:57.671416Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.5","timestamp":"2020-06-04T08:48:57.903478Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:48:58.592 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-59 2020-06-04T08:48:58.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:58.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:58.602 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:58.657 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:58.658 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:58.906 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.56.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.62.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.43.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.37.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:48:58.907 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-59 2020-06-04T08:48:58.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:58.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:58.909 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:58.991 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:58.991 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:59.257 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-06-04T08:48:57.621151Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-04T08:48:57.661671Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5","timestamp":"2020-06-04T08:48:57.661695Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.5","timestamp":"2020-06-04T08:48:57.897532Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:48:59.258 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-59 2020-06-04T08:48:59.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:59.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:59.260 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:59.343 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:59.343 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:59.597 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.56.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.62.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.43.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.37.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:48:59.598 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-59 2020-06-04T08:48:59.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:59.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:59.599 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:59.644 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:59.644 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:48:59.913 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-06-04T08:48:57.562577Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-04T08:48:57.514816Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.43.5","timestamp":"2020-06-04T08:48:57.467330Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.37.5","timestamp":"2020-06-04T08:48:57.616896Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:48:59.914 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-59 2020-06-04T08:48:59.914 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:59.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:48:59.915 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:48:59.992 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:48:59.992 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:00.223 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000001.55.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.57.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.62.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.43.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:49:00.223 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-59 2020-06-04T08:49:00.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:00.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:00.233 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:00.278 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:00.278 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:00.560 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5","timestamp":"2020-06-04T08:48:57.925562Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-06-04T08:48:57.881762Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-04T08:48:57.882104Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5","timestamp":"2020-06-04T08:48:58.161494Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:00.561 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-59 2020-06-04T08:49:00.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:00.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:00.563 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:00.604 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:00.604 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:00.846 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000001.55.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.57.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.62.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.43.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:49:00.847 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-59 2020-06-04T08:49:00.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:00.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:00.848 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:00.897 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:00.898 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:01.166 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5","timestamp":"2020-06-04T08:48:57.881996Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-06-04T08:48:57.881521Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-04T08:48:57.882276Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5","timestamp":"2020-06-04T08:48:58.117467Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:01.167 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-59 2020-06-04T08:49:01.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:01.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:01.169 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:01.212 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:01.213 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:01.459 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.53.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.59.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.52.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.32.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:49:01.459 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-59 2020-06-04T08:49:01.460 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:01.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:01.462 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:01.548 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:01.548 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:01.803 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5","timestamp":"2020-06-04T08:48:57.858265Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5","timestamp":"2020-06-04T08:48:57.901380Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5","timestamp":"2020-06-04T08:48:57.901403Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.32.5","timestamp":"2020-06-04T08:48:58.109543Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:01.804 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-59 2020-06-04T08:49:01.805 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:01.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:01.806 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:01.883 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:01.884 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:02.129 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.53.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.59.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.52.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.32.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:49:02.130 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-59 2020-06-04T08:49:02.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:02.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:02.132 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:02.178 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:02.179 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:02.435 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5","timestamp":"2020-06-04T08:48:57.903541Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5","timestamp":"2020-06-04T08:48:57.903511Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5","timestamp":"2020-06-04T08:48:57.861013Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.32.5","timestamp":"2020-06-04T08:48:58.099489Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:02.435 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-59 2020-06-04T08:49:02.436 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-59 zones={test-zone2, test-zone1} 2020-06-04T08:49:02.464 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:49:02.478 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:49:02.479 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:49:02.480 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:49:02.480 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:49:02.481 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:49:02.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:49:02.491 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:49:02.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:49:02.492 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:49:02.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:02.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:02.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:02.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:02.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:02.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:03.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:03.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:03.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:03.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:03.439 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:03.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:03.659 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-59 zones={test-zone2, test-zone1} 2020-06-04T08:49:03.660 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-59 zones={test-zone3, test-zone1} 2020-06-04T08:49:03.669 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:49:03.676 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:49:03.676 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:49:03.677 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:49:03.677 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:49:03.677 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:49:03.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:49:03.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:49:03.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:49:03.685 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:49:03.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:03.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:03.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:04.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:04.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:04.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:04.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:04.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:04.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:04.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:04.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:04.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:04.871 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-59 zones={test-zone3, test-zone1} 2020-06-04T08:49:04.872 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-59 zones={test-zone3, test-zone2} 2020-06-04T08:49:04.883 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:49:04.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:49:04.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:49:04.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:49:04.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:49:04.893 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:49:04.900 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:49:04.901 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:49:04.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:49:04.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:49:04.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:04.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:05.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:05.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:05.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:05.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:05.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:05.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:05.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:05.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:05.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:05.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:06.081 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-59 zones={test-zone3, test-zone2} 2020-06-04T08:49:06.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:06.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:06.084 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:06.128 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:06.129 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:06.368 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.57.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.42.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.79.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.58.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:49:06.368 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-60 2020-06-04T08:49:06.369 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:06.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:06.370 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:06.415 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:06.416 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:06.669 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-06-04T08:48:58.067406Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5","timestamp":"2020-06-04T08:48:58.303417Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.79.5","timestamp":"2020-06-04T08:48:58.026337Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5","timestamp":"2020-06-04T08:48:58.259450Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:06.670 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-60 2020-06-04T08:49:06.671 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:06.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:06.672 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:06.710 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:06.710 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:06.944 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.57.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.42.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.79.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.58.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:49:06.945 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-60 2020-06-04T08:49:06.946 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:06.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:06.947 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:07.035 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:07.035 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:07.297 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-06-04T08:48:58.057381Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5","timestamp":"2020-06-04T08:48:58.297377Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.79.5","timestamp":"2020-06-04T08:48:58.019624Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5","timestamp":"2020-06-04T08:48:58.257345Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:07.298 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-60 2020-06-04T08:49:07.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:07.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:07.300 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:07.380 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:07.380 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:07.635 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.57.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.42.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.79.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.58.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:49:07.636 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-60 2020-06-04T08:49:07.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:07.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:07.638 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:07.684 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:07.684 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:07.959 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-06-04T08:48:57.766530Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.42.5","timestamp":"2020-06-04T08:48:57.719467Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.79.5","timestamp":"2020-06-04T08:48:57.668361Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.58.5","timestamp":"2020-06-04T08:48:57.815082Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:07.959 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-60 2020-06-04T08:49:07.960 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:07.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:07.961 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:08.006 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:08.006 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:08.209 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000001.43.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.46.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.76.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.57.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:49:08.210 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-60 2020-06-04T08:49:08.211 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:08.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:08.212 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:08.259 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:08.260 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:08.541 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5","timestamp":"2020-06-04T08:48:57.882141Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5","timestamp":"2020-06-04T08:48:57.921481Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5","timestamp":"2020-06-04T08:48:57.880962Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-06-04T08:48:57.921461Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:08.541 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-60 2020-06-04T08:49:08.542 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:08.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:08.544 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:08.592 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:08.593 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:08.833 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000001.43.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.46.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.76.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.57.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:49:08.833 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-60 2020-06-04T08:49:08.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:08.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:08.835 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:08.876 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:08.877 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:09.139 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5","timestamp":"2020-06-04T08:48:57.925477Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5","timestamp":"2020-06-04T08:48:57.921452Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5","timestamp":"2020-06-04T08:48:57.921485Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-06-04T08:48:57.882011Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:09.139 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-60 2020-06-04T08:49:09.140 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:09.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:09.141 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:09.222 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:09.222 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:09.477 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.43.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.46.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.56.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.55.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:49:09.478 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-60 2020-06-04T08:49:09.478 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:09.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:09.480 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:09.565 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:09.565 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:09.837 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5","timestamp":"2020-06-04T08:48:58.337631Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5","timestamp":"2020-06-04T08:48:58.337611Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-06-04T08:48:58.341482Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5","timestamp":"2020-06-04T08:48:58.149618Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:09.838 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-60 2020-06-04T08:49:09.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:09.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:09.841 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:09.884 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:09.884 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:10.133 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.43.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.46.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.56.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.55.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:49:10.133 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-60 2020-06-04T08:49:10.134 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:10.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:10.135 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:10.180 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:10.181 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:10.438 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5","timestamp":"2020-06-04T08:48:58.305623Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5","timestamp":"2020-06-04T08:48:58.343393Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-06-04T08:48:58.347355Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5","timestamp":"2020-06-04T08:48:58.099481Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:10.438 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-60 2020-06-04T08:49:10.439 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-60 zones={test-zone2, test-zone1} 2020-06-04T08:49:10.453 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:49:10.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:49:10.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:49:10.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:49:10.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:49:10.464 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:49:10.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:49:10.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:49:10.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:49:10.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:49:10.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:10.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:10.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:10.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:10.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:10.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:11.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:11.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:11.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:11.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:11.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:11.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:11.653 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-60 zones={test-zone2, test-zone1} 2020-06-04T08:49:11.653 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-60 zones={test-zone3, test-zone1} 2020-06-04T08:49:11.664 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:49:11.671 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:49:11.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:49:11.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:49:11.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:49:11.676 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:49:11.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:49:11.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:49:11.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:49:11.685 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:49:11.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:11.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:11.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:11.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:12.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:12.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:12.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:12.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:12.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:12.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:12.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:12.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:12.884 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-60 zones={test-zone3, test-zone1} 2020-06-04T08:49:12.885 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-60 zones={test-zone3, test-zone2} 2020-06-04T08:49:12.895 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:49:12.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:49:12.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:49:12.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:49:12.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:49:12.911 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:49:12.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:49:12.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:49:12.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:49:12.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:49:12.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:13.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:13.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:13.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:13.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:13.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:13.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:13.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:13.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:13.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:14.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:14.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:14.286 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-60 zones={test-zone3, test-zone2} 2020-06-04T08:49:14.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:14.287 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:14.288 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:14.331 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:14.331 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:14.580 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.40.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.85.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.54.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.76.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:49:14.581 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-61 2020-06-04T08:49:14.581 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:14.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:14.582 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:14.626 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:14.627 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:14.887 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.40.5","timestamp":"2020-06-04T08:48:58.263458Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5","timestamp":"2020-06-04T08:48:58.467401Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5","timestamp":"2020-06-04T08:48:58.503601Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5","timestamp":"2020-06-04T08:48:58.503566Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:14.888 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-61 2020-06-04T08:49:14.888 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:14.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:14.889 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:14.934 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:14.934 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:15.182 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.40.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.85.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.54.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.76.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:49:15.182 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-61 2020-06-04T08:49:15.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:15.183 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:15.184 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:15.268 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:15.269 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:15.539 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.40.5","timestamp":"2020-06-04T08:48:58.261336Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5","timestamp":"2020-06-04T08:48:58.505516Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5","timestamp":"2020-06-04T08:48:58.457385Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5","timestamp":"2020-06-04T08:48:58.457404Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:15.539 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-61 2020-06-04T08:49:15.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:15.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:15.541 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:15.582 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:15.583 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:15.818 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.40.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.85.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.54.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.76.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:49:15.819 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-61 2020-06-04T08:49:15.821 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:15.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:15.825 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:15.875 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:15.876 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:16.151 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.40.5","timestamp":"2020-06-04T08:48:57.968778Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.85.5","timestamp":"2020-06-04T08:48:57.914616Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.54.5","timestamp":"2020-06-04T08:48:57.860894Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.76.5","timestamp":"2020-06-04T08:48:58.019441Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:16.151 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-61 2020-06-04T08:49:16.152 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:16.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:16.154 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:16.239 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:16.239 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:16.480 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000001.33.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.82.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.45.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.57.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:49:16.480 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-61 2020-06-04T08:49:16.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:16.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:16.482 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:16.534 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:16.534 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:16.801 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.33.5","timestamp":"2020-06-04T08:48:58.321543Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.82.5","timestamp":"2020-06-04T08:48:58.321519Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5","timestamp":"2020-06-04T08:48:58.321558Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-06-04T08:48:58.321551Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:16.801 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-61 2020-06-04T08:49:16.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:16.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:16.803 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:16.858 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:16.858 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:17.100 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000001.33.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.82.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.45.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.57.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:49:17.100 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-61 2020-06-04T08:49:17.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:17.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:17.102 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:17.151 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:17.152 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:17.424 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.33.5","timestamp":"2020-06-04T08:48:58.313632Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.82.5","timestamp":"2020-06-04T08:48:58.361387Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5","timestamp":"2020-06-04T08:48:58.313614Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-06-04T08:48:58.273633Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:17.427 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-61 2020-06-04T08:49:17.428 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:17.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:17.431 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:17.512 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:17.513 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:17.753 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.29.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.99.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.56.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.52.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:49:17.754 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-61 2020-06-04T08:49:17.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:17.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:17.755 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:17.839 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:17.840 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:18.103 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.29.5","timestamp":"2020-06-04T08:48:58.050313Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.99.5","timestamp":"2020-06-04T08:48:58.145486Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-06-04T08:48:58.093849Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5","timestamp":"2020-06-04T08:48:58.145452Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:18.104 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-61 2020-06-04T08:49:18.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:18.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:18.105 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:18.194 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:18.194 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:18.432 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.29.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.99.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.56.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.52.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:49:18.432 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-61 2020-06-04T08:49:18.433 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:18.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:18.434 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:18.482 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:18.482 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:18.742 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.29.5","timestamp":"2020-06-04T08:48:58.058442Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.99.5","timestamp":"2020-06-04T08:48:58.099455Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-06-04T08:48:58.058311Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5","timestamp":"2020-06-04T08:48:58.058472Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:18.742 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-61 2020-06-04T08:49:18.743 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-61 zones={test-zone2, test-zone1} 2020-06-04T08:49:18.759 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:49:18.765 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:49:18.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:49:18.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:49:18.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:49:18.768 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:49:18.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:49:18.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:49:18.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:49:18.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:49:18.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:18.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:18.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:19.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:19.163 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:19.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:19.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:19.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:19.549 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:19.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:19.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:19.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:19.956 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-61 zones={test-zone2, test-zone1} 2020-06-04T08:49:19.957 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-61 zones={test-zone3, test-zone1} 2020-06-04T08:49:19.966 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:49:19.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:49:19.974 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:49:19.975 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:49:19.975 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:49:19.976 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:49:19.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:49:19.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:49:19.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:49:19.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:49:19.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:20.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:20.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:20.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:20.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:20.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:20.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:20.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:20.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:20.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:20.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:21.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:21.191 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-61 zones={test-zone3, test-zone1} 2020-06-04T08:49:21.192 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-61 zones={test-zone3, test-zone2} 2020-06-04T08:49:21.202 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:49:21.208 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:49:21.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:49:21.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:49:21.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:49:21.211 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:49:21.217 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:49:21.218 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:49:21.218 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:49:21.218 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:49:21.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:21.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:21.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:21.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:21.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:21.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:21.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:21.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:21.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:22.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:22.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:22.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:22.382 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-61 zones={test-zone3, test-zone2} 2020-06-04T08:49:22.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'gpxthc-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:49:22.383 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'gpxthc-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:49:22.384 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket gpxthc-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:49:22.426 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:22.427 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:23.890 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'gpxthc-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:49:23.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'gpxthc-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:49:23.894 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket gpxthc-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:49:23.976 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:23.976 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:24.312 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'gpxthc-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:49:24.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'gpxthc-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:49:24.314 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket gpxthc-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:49:24.398 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:24.399 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:24.731 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'gpxthc-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:49:24.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'gpxthc-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:49:24.734 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket gpxthc-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:49:24.810 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:24.811 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:25.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'gpxthc-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:49:25.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'gpxthc-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:49:25.139 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket gpxthc-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:49:25.221 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:25.222 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:25.553 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'gpxthc-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:49:25.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'gpxthc-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:49:25.555 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket gpxthc-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:49:25.642 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:25.642 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:26.574 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:26.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:26.576 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:26.619 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:26.620 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:26.848 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000006.72.5"},{"key":1,"val":"00000000004.58.1"},{"key":2,"val":"00000000006.68.5"},{"key":3,"val":"00000000004.39.1"},{"key":4,"val":"00000000006.53.5"},{"key":5,"val":"00000000004.44.1"},{"key":6,"val":"00000000004.92.1"},{"key":7,"val":"00000000006.43.5"},{"key":8,"val":"00000000004.83.1"},{"key":9,"val":"00000000004.93.1"},{"key":10,"val":"00000000004.44.1"}]} 2020-06-04T08:49:26.849 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-59 2020-06-04T08:49:26.849 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:26.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:26.850 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:26.895 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:26.896 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:27.410 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.72.5","timestamp":"2020-06-04T08:49:26.082920Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.68.5","timestamp":"2020-06-04T08:49:26.303500Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1","timestamp":"2020-06-04T08:49:26.295478Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.53.5","timestamp":"2020-06-04T08:49:26.091602Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1","timestamp":"2020-06-04T08:49:26.303474Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.43.5","timestamp":"2020-06-04T08:49:26.299411Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.83.1","timestamp":"2020-06-04T08:49:26.095517Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.44.1","timestamp":"2020-06-04T08:49:26.295444Z"}}}] 2020-06-04T08:49:27.410 WARNING:rgw_multi.tests:bucket gpxthc-59 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.58.1 target=, shard=5 master=00000000004.44.1 target=, shard=9 master=00000000004.93.1 target= 2020-06-04T08:49:32.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:32.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:32.414 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:32.469 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:32.469 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:32.685 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.72.5","timestamp":"2020-06-04T08:49:26.082920Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1","timestamp":"2020-06-04T08:49:32.507589Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.68.5","timestamp":"2020-06-04T08:49:26.303500Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1","timestamp":"2020-06-04T08:49:26.295478Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.53.5","timestamp":"2020-06-04T08:49:26.091602Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.44.1","timestamp":"2020-06-04T08:49:32.507625Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1","timestamp":"2020-06-04T08:49:26.303474Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.43.5","timestamp":"2020-06-04T08:49:26.299411Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.83.1","timestamp":"2020-06-04T08:49:26.095517Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1","timestamp":"2020-06-04T08:49:32.458286Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.44.1","timestamp":"2020-06-04T08:49:26.295444Z"}}}] 2020-06-04T08:49:32.686 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-59 2020-06-04T08:49:32.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:32.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:32.687 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:32.730 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:32.731 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:32.968 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000006.72.5"},{"key":1,"val":"00000000004.58.1"},{"key":2,"val":"00000000006.68.5"},{"key":3,"val":"00000000004.39.1"},{"key":4,"val":"00000000006.53.5"},{"key":5,"val":"00000000004.44.1"},{"key":6,"val":"00000000004.92.1"},{"key":7,"val":"00000000006.43.5"},{"key":8,"val":"00000000004.83.1"},{"key":9,"val":"00000000004.93.1"},{"key":10,"val":"00000000004.44.1"}]} 2020-06-04T08:49:32.968 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-59 2020-06-04T08:49:32.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:32.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:32.970 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:33.060 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:33.061 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:33.336 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.72.5","timestamp":"2020-06-04T08:49:26.082920Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1","timestamp":"2020-06-04T08:49:32.478561Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.68.5","timestamp":"2020-06-04T08:49:26.370038Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1","timestamp":"2020-06-04T08:49:26.269791Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.53.5","timestamp":"2020-06-04T08:49:26.046098Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.44.1","timestamp":"2020-06-04T08:49:32.521393Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1","timestamp":"2020-06-04T08:49:26.325996Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.43.5","timestamp":"2020-06-04T08:49:26.321414Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.83.1","timestamp":"2020-06-04T08:49:26.081430Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1","timestamp":"2020-06-04T08:49:32.481190Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.44.1","timestamp":"2020-06-04T08:49:26.325983Z"}}}] 2020-06-04T08:49:33.337 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-59 2020-06-04T08:49:33.338 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:33.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:33.341 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:33.384 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:33.385 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:33.635 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000006.72.5"},{"key":1,"val":"00000000004.58.1"},{"key":2,"val":"00000000006.68.5"},{"key":3,"val":"00000000004.39.1"},{"key":4,"val":"00000000006.53.5"},{"key":5,"val":"00000000004.44.1"},{"key":6,"val":"00000000004.92.1"},{"key":7,"val":"00000000006.43.5"},{"key":8,"val":"00000000004.83.1"},{"key":9,"val":"00000000004.93.1"},{"key":10,"val":"00000000004.44.1"}]} 2020-06-04T08:49:33.635 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-59 2020-06-04T08:49:33.636 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:33.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:33.637 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:33.684 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:33.684 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:33.950 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.72.5","timestamp":"2020-06-04T08:49:26.082920Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.58.1","timestamp":"2020-06-04T08:49:24.259430Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.68.5","timestamp":"2020-06-04T08:49:26.031828Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1","timestamp":"2020-06-04T08:49:24.259597Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.53.5","timestamp":"2020-06-04T08:49:25.978397Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.44.1","timestamp":"2020-06-04T08:49:24.259739Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.92.1","timestamp":"2020-06-04T08:49:24.260338Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.43.5","timestamp":"2020-06-04T08:49:26.129066Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.83.1","timestamp":"2020-06-04T08:49:24.260349Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.93.1","timestamp":"2020-06-04T08:49:24.262423Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.44.1","timestamp":"2020-06-04T08:49:24.262943Z"}}}] 2020-06-04T08:49:33.951 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-59 2020-06-04T08:49:33.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:33.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:33.952 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:33.993 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:33.994 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:34.228 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000002.59.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.59.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.64.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.45.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:49:34.229 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-59 2020-06-04T08:49:34.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:34.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:34.232 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:34.279 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:34.279 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:34.562 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5","timestamp":"2020-06-04T08:48:57.925562Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-06-04T08:48:57.881762Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-04T08:48:57.882104Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5","timestamp":"2020-06-04T08:48:58.161494Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:34.563 WARNING:rgw_multi.tests:bucket gpxthc-59 zone test-zone1 behind zone test-zone2: shard=0 master=00000000002.59.5 target=00000000001.55.5, shard=2 master=00000000002.59.5 target=00000000001.57.5, shard=4 master=00000000002.64.5 target=00000000001.62.5, shard=7 master=00000000002.45.5 target=00000000001.43.5 2020-06-04T08:49:39.564 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:39.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:39.566 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:39.610 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:39.610 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:39.877 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5","timestamp":"2020-06-04T08:48:57.925562Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-06-04T08:48:57.881762Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-04T08:48:57.882104Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5","timestamp":"2020-06-04T08:48:58.161494Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:39.878 WARNING:rgw_multi.tests:bucket gpxthc-59 zone test-zone1 behind zone test-zone2: shard=0 master=00000000002.59.5 target=00000000001.55.5, shard=2 master=00000000002.59.5 target=00000000001.57.5, shard=4 master=00000000002.64.5 target=00000000001.62.5, shard=7 master=00000000002.45.5 target=00000000001.43.5 2020-06-04T08:49:44.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:44.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:44.881 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:44.950 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:44.951 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:45.226 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5","timestamp":"2020-06-04T08:48:57.925562Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-06-04T08:48:57.881762Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-04T08:48:57.882104Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5","timestamp":"2020-06-04T08:48:58.161494Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:45.227 WARNING:rgw_multi.tests:bucket gpxthc-59 zone test-zone1 behind zone test-zone2: shard=0 master=00000000002.59.5 target=00000000001.55.5, shard=2 master=00000000002.59.5 target=00000000001.57.5, shard=4 master=00000000002.64.5 target=00000000001.62.5, shard=7 master=00000000002.45.5 target=00000000001.43.5 2020-06-04T08:49:50.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:50.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:50.231 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:50.277 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:50.277 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:50.552 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5","timestamp":"2020-06-04T08:48:57.925562Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5","timestamp":"2020-06-04T08:48:57.881762Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-04T08:48:57.882104Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5","timestamp":"2020-06-04T08:48:58.161494Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:50.552 WARNING:rgw_multi.tests:bucket gpxthc-59 zone test-zone1 behind zone test-zone2: shard=0 master=00000000002.59.5 target=00000000001.55.5, shard=2 master=00000000002.59.5 target=00000000001.57.5, shard=4 master=00000000002.64.5 target=00000000001.62.5, shard=7 master=00000000002.45.5 target=00000000001.43.5 2020-06-04T08:49:55.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:55.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:55.555 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:55.603 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:55.603 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:55.872 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.59.5","timestamp":"2020-06-04T08:49:26.082920Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.59.5","timestamp":"2020-06-04T08:49:26.031828Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.64.5","timestamp":"2020-06-04T08:49:25.978397Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.45.5","timestamp":"2020-06-04T08:49:26.129066Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:55.873 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-59 2020-06-04T08:49:55.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:55.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:55.875 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:55.916 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:55.916 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:56.146 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000002.59.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.59.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.64.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.45.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:49:56.147 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-59 2020-06-04T08:49:56.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:56.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:56.149 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:56.194 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:56.195 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:56.454 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.59.5","timestamp":"2020-06-04T08:49:26.082920Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.59.5","timestamp":"2020-06-04T08:49:26.031828Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.64.5","timestamp":"2020-06-04T08:49:25.978397Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.45.5","timestamp":"2020-06-04T08:49:26.129066Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:56.454 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-59 2020-06-04T08:49:56.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:56.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:56.456 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:56.537 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:56.537 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:56.766 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000002.57.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.61.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.54.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.34.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:49:56.767 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-59 2020-06-04T08:49:56.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:56.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:56.772 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:56.854 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:56.854 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:57.121 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.57.5","timestamp":"2020-06-04T08:49:26.082920Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.61.5","timestamp":"2020-06-04T08:49:26.031828Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.54.5","timestamp":"2020-06-04T08:49:25.978397Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.34.5","timestamp":"2020-06-04T08:49:26.129066Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:57.122 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-59 2020-06-04T08:49:57.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:57.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:57.124 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:57.165 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:57.166 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:57.407 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000002.57.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.61.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.54.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.34.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:49:57.407 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-59 2020-06-04T08:49:57.408 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:57.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:49:57.409 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:49:57.454 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:49:57.455 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:49:57.710 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.57.5","timestamp":"2020-06-04T08:49:26.082920Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.61.5","timestamp":"2020-06-04T08:49:26.031828Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.54.5","timestamp":"2020-06-04T08:49:25.978397Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.34.5","timestamp":"2020-06-04T08:49:26.129066Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:49:57.711 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-59 2020-06-04T08:49:57.712 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-59 zones={test-zone2, test-zone1} 2020-06-04T08:49:57.725 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:49:57.733 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:49:57.734 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:49:57.734 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:49:57.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:49:57.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:49:57.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:49:57.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:49:57.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:49:57.737 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:49:57.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:49:57.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:49:57.747 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:49:57.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:49:57.748 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:49:57.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:49:57.749 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:49:57.750 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:49:57.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:57.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:57.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:49:58.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:58.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:58.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:49:58.325 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:58.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:58.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:49:58.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:58.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:58.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:49:58.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:49:59.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:49:59.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:49:59.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:49:59.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:49:59.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:49:59.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:49:59.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:49:59.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:49:59.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:49:59.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:49:59.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:00.077 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-59 zones={test-zone2, test-zone1} 2020-06-04T08:50:00.077 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-59 zones={test-zone3, test-zone1} 2020-06-04T08:50:00.086 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:50:00.094 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:00.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:00.095 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:00.096 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:00.097 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:50:00.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:50:00.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:50:00.099 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:50:00.100 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:50:00.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:00.109 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:00.110 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:00.111 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:00.111 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:50:00.111 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:50:00.112 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:50:00.112 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:50:00.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:00.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:00.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:00.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:00.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:00.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:00.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:00.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:00.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:01.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:01.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:01.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:01.304 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:01.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:01.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:01.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:01.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:01.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:01.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:01.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:02.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:02.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:02.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:02.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:02.460 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-59 zones={test-zone3, test-zone1} 2020-06-04T08:50:02.460 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-59 zones={test-zone3, test-zone2} 2020-06-04T08:50:02.473 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:50:02.481 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:02.482 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:02.482 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:02.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:02.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:50:02.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:50:02.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:50:02.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:50:02.485 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:50:02.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:02.493 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:02.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:02.494 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:02.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:50:02.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:50:02.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:50:02.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:50:02.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:02.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:02.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:02.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:02.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:02.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:03.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:03.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:03.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:03.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:03.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:03.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:03.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:03.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:03.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:03.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:03.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:04.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:04.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:04.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:04.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:04.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:04.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:04.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:04.810 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-59 zones={test-zone3, test-zone2} 2020-06-04T08:50:04.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:04.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:04.812 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:04.855 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:04.855 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:05.119 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000006.63.5"},{"key":1,"val":"00000000004.48.1"},{"key":2,"val":"00000000006.48.5"},{"key":3,"val":"00000000004.54.1"},{"key":4,"val":"00000000006.93.5"},{"key":5,"val":"00000000004.69.1"},{"key":6,"val":"00000000004.71.1"},{"key":7,"val":"00000000006.74.5"},{"key":8,"val":"00000000004.55.1"},{"key":9,"val":"00000000004.97.1"},{"key":10,"val":"00000000004.48.1"}]} 2020-06-04T08:50:05.120 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-60 2020-06-04T08:50:05.120 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:05.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:05.122 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:05.168 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:05.168 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:05.421 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.63.5","timestamp":"2020-06-04T08:49:26.280888Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1","timestamp":"2020-06-04T08:49:32.462401Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.48.5","timestamp":"2020-06-04T08:49:26.234354Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1","timestamp":"2020-06-04T08:49:26.269231Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.93.5","timestamp":"2020-06-04T08:49:26.184760Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.1","timestamp":"2020-06-04T08:49:32.459463Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1","timestamp":"2020-06-04T08:49:26.269210Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.74.5","timestamp":"2020-06-04T08:49:26.329795Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1","timestamp":"2020-06-04T08:49:26.053965Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.97.1","timestamp":"2020-06-04T08:49:32.460373Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1","timestamp":"2020-06-04T08:49:26.262954Z"}}}] 2020-06-04T08:50:05.422 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-60 2020-06-04T08:50:05.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:05.423 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:05.424 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:05.466 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:05.703 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:05.720 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000006.63.5"},{"key":1,"val":"00000000004.48.1"},{"key":2,"val":"00000000006.48.5"},{"key":3,"val":"00000000004.54.1"},{"key":4,"val":"00000000006.93.5"},{"key":5,"val":"00000000004.69.1"},{"key":6,"val":"00000000004.71.1"},{"key":7,"val":"00000000006.74.5"},{"key":8,"val":"00000000004.55.1"},{"key":9,"val":"00000000004.97.1"},{"key":10,"val":"00000000004.48.1"}]} 2020-06-04T08:50:05.721 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-60 2020-06-04T08:50:05.721 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:05.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:05.723 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:05.803 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:05.804 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:06.062 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.63.5","timestamp":"2020-06-04T08:49:26.280888Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1","timestamp":"2020-06-04T08:49:32.525536Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.48.5","timestamp":"2020-06-04T08:49:26.234354Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1","timestamp":"2020-06-04T08:49:26.277579Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.93.5","timestamp":"2020-06-04T08:49:26.184760Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.1","timestamp":"2020-06-04T08:49:32.525563Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1","timestamp":"2020-06-04T08:49:26.333581Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.74.5","timestamp":"2020-06-04T08:49:26.329795Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1","timestamp":"2020-06-04T08:49:26.049899Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.97.1","timestamp":"2020-06-04T08:49:32.525555Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1","timestamp":"2020-06-04T08:49:26.286763Z"}}}] 2020-06-04T08:50:06.063 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-60 2020-06-04T08:50:06.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:06.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:06.064 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:06.151 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:06.152 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:06.405 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000006.63.5"},{"key":1,"val":"00000000004.48.1"},{"key":2,"val":"00000000006.48.5"},{"key":3,"val":"00000000004.54.1"},{"key":4,"val":"00000000006.93.5"},{"key":5,"val":"00000000004.69.1"},{"key":6,"val":"00000000004.71.1"},{"key":7,"val":"00000000006.74.5"},{"key":8,"val":"00000000004.55.1"},{"key":9,"val":"00000000004.97.1"},{"key":10,"val":"00000000004.48.1"}]} 2020-06-04T08:50:06.405 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-60 2020-06-04T08:50:06.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:06.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:06.407 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:06.451 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:06.451 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:06.717 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.63.5","timestamp":"2020-06-04T08:49:26.280888Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1","timestamp":"2020-06-04T08:49:25.087190Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.48.5","timestamp":"2020-06-04T08:49:26.234354Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1","timestamp":"2020-06-04T08:49:25.087416Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.93.5","timestamp":"2020-06-04T08:49:26.184760Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.1","timestamp":"2020-06-04T08:49:25.088782Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1","timestamp":"2020-06-04T08:49:25.088593Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.74.5","timestamp":"2020-06-04T08:49:26.329795Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1","timestamp":"2020-06-04T08:49:25.090332Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.97.1","timestamp":"2020-06-04T08:49:25.091215Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1","timestamp":"2020-06-04T08:49:25.086859Z"}}}] 2020-06-04T08:50:06.718 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-60 2020-06-04T08:50:06.719 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:06.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:06.720 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:06.802 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:06.802 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:07.038 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000002.45.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.48.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.78.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.61.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:07.039 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-60 2020-06-04T08:50:07.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:07.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:07.041 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:07.097 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:07.098 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:07.383 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.45.5","timestamp":"2020-06-04T08:49:26.280888Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.48.5","timestamp":"2020-06-04T08:49:26.234354Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.78.5","timestamp":"2020-06-04T08:49:26.184760Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.61.5","timestamp":"2020-06-04T08:49:26.329795Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:07.384 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-60 2020-06-04T08:50:07.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:07.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:07.386 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:07.427 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:07.428 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:07.656 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000002.45.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.48.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.78.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.61.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:07.657 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-60 2020-06-04T08:50:07.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:07.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:07.659 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:07.702 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:07.703 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:07.971 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.45.5","timestamp":"2020-06-04T08:49:26.280888Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.48.5","timestamp":"2020-06-04T08:49:26.234354Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.78.5","timestamp":"2020-06-04T08:49:26.184760Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.61.5","timestamp":"2020-06-04T08:49:26.329795Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:07.972 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-60 2020-06-04T08:50:07.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:07.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:07.973 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:08.058 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:08.058 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:08.297 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000003.49.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.50.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.64.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.65.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:08.298 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-60 2020-06-04T08:50:08.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:08.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:08.301 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:08.383 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:08.383 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:08.658 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.49.5","timestamp":"2020-06-04T08:49:26.280888Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.50.5","timestamp":"2020-06-04T08:49:26.234354Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.64.5","timestamp":"2020-06-04T08:49:26.184760Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.65.5","timestamp":"2020-06-04T08:49:26.329795Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:08.658 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-60 2020-06-04T08:50:08.659 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:08.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:08.660 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:08.744 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:08.745 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:08.982 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000003.49.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.50.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.64.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.65.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:08.983 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-60 2020-06-04T08:50:08.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:08.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:08.985 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:09.028 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:09.028 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:09.287 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.49.5","timestamp":"2020-06-04T08:49:26.280888Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.50.5","timestamp":"2020-06-04T08:49:26.234354Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.64.5","timestamp":"2020-06-04T08:49:26.184760Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.65.5","timestamp":"2020-06-04T08:49:26.329795Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:09.287 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-60 2020-06-04T08:50:09.288 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-60 zones={test-zone2, test-zone1} 2020-06-04T08:50:09.299 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:50:09.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:09.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:09.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:09.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:09.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:50:09.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:50:09.309 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:50:09.310 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:50:09.310 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:50:09.320 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:09.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:09.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:09.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:09.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:50:09.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:50:09.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:50:09.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:50:09.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:09.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:09.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:09.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:09.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:09.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:09.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:09.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:10.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:10.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:10.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:10.349 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:10.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:10.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:10.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:10.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:10.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:11.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:11.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:11.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:11.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:11.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:11.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:11.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:11.729 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-60 zones={test-zone2, test-zone1} 2020-06-04T08:50:11.730 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-60 zones={test-zone3, test-zone1} 2020-06-04T08:50:11.745 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:50:11.753 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:11.754 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:11.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:11.755 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:11.756 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:50:11.756 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:50:11.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:50:11.757 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:50:11.757 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:50:11.766 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:11.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:11.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:11.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:11.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:50:11.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:50:11.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:50:11.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:50:11.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:11.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:11.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:12.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:12.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:12.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:12.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:12.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:12.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:12.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:12.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:12.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:12.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:12.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:13.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:13.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:13.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:13.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:13.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:13.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:13.700 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:13.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:13.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:13.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:14.103 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-60 zones={test-zone3, test-zone1} 2020-06-04T08:50:14.104 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-60 zones={test-zone3, test-zone2} 2020-06-04T08:50:14.116 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:50:14.129 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:14.130 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:14.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:14.131 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:14.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:50:14.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:50:14.133 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:50:14.134 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:50:14.134 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:50:14.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:14.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:14.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:14.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:14.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:50:14.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:50:14.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:50:14.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:50:14.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:14.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:14.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:14.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:14.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:14.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:14.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:14.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:14.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:15.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:15.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:15.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:15.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:15.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:15.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:15.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:15.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:15.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:15.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:15.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:16.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:16.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:16.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:16.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:16.450 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-60 zones={test-zone3, test-zone2} 2020-06-04T08:50:16.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:16.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:16.452 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:16.495 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:16.495 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:16.743 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000006.54.5"},{"key":1,"val":"00000000004.48.1"},{"key":2,"val":"00000000006.103.5"},{"key":3,"val":"00000000004.75.1"},{"key":4,"val":"00000000006.64.5"},{"key":5,"val":"00000000004.59.1"},{"key":6,"val":"00000000004.53.1"},{"key":7,"val":"00000000006.82.5"},{"key":8,"val":"00000000004.91.1"},{"key":9,"val":"00000000004.51.1"},{"key":10,"val":"00000000004.55.1"}]} 2020-06-04T08:50:16.744 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-61 2020-06-04T08:50:16.744 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:16.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:16.745 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:16.789 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:16.790 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:17.043 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.54.5","timestamp":"2020-06-04T08:49:26.483087Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1","timestamp":"2020-06-04T08:49:32.495473Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.103.5","timestamp":"2020-06-04T08:49:26.434482Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1","timestamp":"2020-06-04T08:49:26.311753Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.64.5","timestamp":"2020-06-04T08:49:26.374460Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.59.1","timestamp":"2020-06-04T08:49:32.462381Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1","timestamp":"2020-06-04T08:49:26.260369Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.82.5","timestamp":"2020-06-04T08:49:26.531279Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.91.1","timestamp":"2020-06-04T08:49:26.053992Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1","timestamp":"2020-06-04T08:49:32.499448Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1","timestamp":"2020-06-04T08:49:26.263687Z"}}}] 2020-06-04T08:50:17.044 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-61 2020-06-04T08:50:17.044 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:17.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:17.045 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:17.087 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:17.087 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:17.338 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000006.54.5"},{"key":1,"val":"00000000004.48.1"},{"key":2,"val":"00000000006.103.5"},{"key":3,"val":"00000000004.75.1"},{"key":4,"val":"00000000006.64.5"},{"key":5,"val":"00000000004.59.1"},{"key":6,"val":"00000000004.53.1"},{"key":7,"val":"00000000006.82.5"},{"key":8,"val":"00000000004.91.1"},{"key":9,"val":"00000000004.51.1"},{"key":10,"val":"00000000004.55.1"}]} 2020-06-04T08:50:17.338 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-61 2020-06-04T08:50:17.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:17.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:17.340 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:17.426 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:17.427 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:17.688 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.54.5","timestamp":"2020-06-04T08:49:26.483087Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1","timestamp":"2020-06-04T08:49:32.481040Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.103.5","timestamp":"2020-06-04T08:49:26.434482Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1","timestamp":"2020-06-04T08:49:26.277546Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.64.5","timestamp":"2020-06-04T08:49:26.374460Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.59.1","timestamp":"2020-06-04T08:49:32.481161Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1","timestamp":"2020-06-04T08:49:26.290778Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.82.5","timestamp":"2020-06-04T08:49:26.531279Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.91.1","timestamp":"2020-06-04T08:49:26.049945Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1","timestamp":"2020-06-04T08:49:32.479805Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1","timestamp":"2020-06-04T08:49:26.333592Z"}}}] 2020-06-04T08:50:17.688 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-61 2020-06-04T08:50:17.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:17.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:17.690 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:17.733 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:17.734 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:17.984 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000006.54.5"},{"key":1,"val":"00000000004.48.1"},{"key":2,"val":"00000000006.103.5"},{"key":3,"val":"00000000004.75.1"},{"key":4,"val":"00000000006.64.5"},{"key":5,"val":"00000000004.59.1"},{"key":6,"val":"00000000004.53.1"},{"key":7,"val":"00000000006.82.5"},{"key":8,"val":"00000000004.91.1"},{"key":9,"val":"00000000004.51.1"},{"key":10,"val":"00000000004.55.1"}]} 2020-06-04T08:50:17.985 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-61 2020-06-04T08:50:17.985 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:17.986 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:17.987 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:18.031 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:18.032 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:18.291 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.54.5","timestamp":"2020-06-04T08:49:26.483087Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.48.1","timestamp":"2020-06-04T08:49:25.924621Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.103.5","timestamp":"2020-06-04T08:49:26.434482Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.1","timestamp":"2020-06-04T08:49:25.925022Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.64.5","timestamp":"2020-06-04T08:49:26.374460Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.59.1","timestamp":"2020-06-04T08:49:25.927997Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1","timestamp":"2020-06-04T08:49:25.924146Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.82.5","timestamp":"2020-06-04T08:49:26.531279Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.91.1","timestamp":"2020-06-04T08:49:25.928094Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1","timestamp":"2020-06-04T08:49:25.924374Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1","timestamp":"2020-06-04T08:49:25.927603Z"}}}] 2020-06-04T08:50:18.292 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-61 2020-06-04T08:50:18.292 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:18.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:18.294 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:18.385 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:18.386 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:18.612 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000002.41.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.92.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.51.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.61.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:18.613 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-61 2020-06-04T08:50:18.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:18.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:18.615 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:18.662 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:18.662 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:18.941 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.41.5","timestamp":"2020-06-04T08:49:26.483087Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.92.5","timestamp":"2020-06-04T08:49:26.434482Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.51.5","timestamp":"2020-06-04T08:49:26.374460Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.61.5","timestamp":"2020-06-04T08:49:26.531279Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:18.942 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-61 2020-06-04T08:50:18.943 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:18.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:18.944 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:18.986 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:18.986 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:19.229 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000002.41.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.92.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.51.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.61.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:19.230 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-61 2020-06-04T08:50:19.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:19.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:19.231 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:19.276 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:19.277 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:19.549 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.41.5","timestamp":"2020-06-04T08:49:26.483087Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.92.5","timestamp":"2020-06-04T08:49:26.434482Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.51.5","timestamp":"2020-06-04T08:49:26.374460Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.61.5","timestamp":"2020-06-04T08:49:26.531279Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:19.550 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-61 2020-06-04T08:50:19.550 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:19.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:19.552 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:19.631 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:19.632 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:19.876 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000002.31.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.101.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.58.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.54.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:19.876 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-61 2020-06-04T08:50:19.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:19.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:19.878 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:19.963 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:19.963 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:20.233 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.31.5","timestamp":"2020-06-04T08:49:26.483087Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.101.5","timestamp":"2020-06-04T08:49:26.434482Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.58.5","timestamp":"2020-06-04T08:49:26.374460Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.54.5","timestamp":"2020-06-04T08:49:26.531279Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:20.233 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-61 2020-06-04T08:50:20.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:20.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:20.235 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:20.275 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:20.276 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:20.529 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000002.31.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.101.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.58.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.54.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:20.530 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-61 2020-06-04T08:50:20.531 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:20.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:20.532 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:20.575 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:20.576 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:20.830 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.31.5","timestamp":"2020-06-04T08:49:26.483087Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.101.5","timestamp":"2020-06-04T08:49:26.434482Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.58.5","timestamp":"2020-06-04T08:49:26.374460Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.54.5","timestamp":"2020-06-04T08:49:26.531279Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:20.831 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-61 2020-06-04T08:50:20.831 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-61 zones={test-zone2, test-zone1} 2020-06-04T08:50:20.844 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:50:20.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:20.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:20.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:20.852 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:20.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:50:20.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:50:20.853 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:50:20.854 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:50:20.854 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:50:20.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:20.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:20.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:20.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:20.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:50:20.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:50:20.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:50:20.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:50:20.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:20.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:21.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:21.180 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:21.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:21.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:21.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:21.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:21.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:21.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:21.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:21.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:22.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:22.101 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:22.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:22.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:22.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:22.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:22.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:22.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:22.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:22.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:22.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:23.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:23.187 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-61 zones={test-zone2, test-zone1} 2020-06-04T08:50:23.189 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-61 zones={test-zone3, test-zone1} 2020-06-04T08:50:23.196 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:50:23.202 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:23.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:23.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:23.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:23.204 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:50:23.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:50:23.205 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:50:23.206 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:50:23.207 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:50:23.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:23.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:23.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:23.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:23.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:50:23.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:50:23.216 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:50:23.217 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:50:23.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:23.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:23.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:23.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:23.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:23.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:23.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:23.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:23.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:24.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:24.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:24.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:24.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:24.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:24.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:24.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:24.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:24.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:24.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:24.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:25.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:25.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:25.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:25.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:25.499 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-61 zones={test-zone3, test-zone1} 2020-06-04T08:50:25.499 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-61 zones={test-zone3, test-zone2} 2020-06-04T08:50:25.511 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:50:25.518 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:25.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:25.519 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:25.520 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:25.520 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:50:25.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:50:25.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:50:25.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:50:25.522 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:50:25.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:25.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:25.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:25.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:25.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:50:25.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:50:25.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:50:25.532 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:50:25.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:25.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:25.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:25.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:25.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:26.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:26.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:26.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:26.272 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:26.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:26.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:26.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:26.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:27.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:27.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:50:27.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:27.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:27.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:50:27.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:28.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:28.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:50:28.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:28.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:28.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:50:28.539 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-61 zones={test-zone3, test-zone2} 2020-06-04T08:50:28.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:28.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:28.541 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:28.582 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:28.582 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:28.852 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260565.934456_240.3","last_update":"2020-06-04T08:49:25.934456Z"},{"marker":"1_1591260592.648943_524.1","last_update":"2020-06-04T08:49:52.648943Z"},{"marker":"1_1591260592.683590_339.1","last_update":"2020-06-04T08:49:52.683590Z"},{"marker":"1_1591260592.527939_578.1","last_update":"2020-06-04T08:49:52.527939Z"}] 2020-06-04T08:50:28.853 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-06-04T08:50:28.854 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:28.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:28.863 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:28.906 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:28.907 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:29.137 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11207024744551397021},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260565.934456_240.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:49:25.934456Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591260592.648943_524.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:49:52.648943Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591260592.683590_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:49:52.683590Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260592.527939_578.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:49:52.527939Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:50:29.138 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-06-04T08:50:29.139 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:29.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:29.140 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:29.181 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:29.181 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:29.440 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260565.934456_240.3","last_update":"2020-06-04T08:49:25.934456Z"},{"marker":"1_1591260592.648943_524.1","last_update":"2020-06-04T08:49:52.648943Z"},{"marker":"1_1591260592.683590_339.1","last_update":"2020-06-04T08:49:52.683590Z"},{"marker":"1_1591260592.527939_578.1","last_update":"2020-06-04T08:49:52.527939Z"}] 2020-06-04T08:50:29.440 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-06-04T08:50:29.441 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:29.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:29.442 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:29.522 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:29.523 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:29.789 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2627143167498148538},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260565.934456_240.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:49:25.934456Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591260592.648943_524.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:49:52.648943Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591260592.683590_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:49:52.683590Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260592.527939_578.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:49:52.527939Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:50:29.789 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-06-04T08:50:29.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:29.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:29.791 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:29.870 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:29.870 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:30.127 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260565.934456_240.3","last_update":"2020-06-04T08:49:25.934456Z"},{"marker":"1_1591260592.648943_524.1","last_update":"2020-06-04T08:49:52.648943Z"},{"marker":"1_1591260592.683590_339.1","last_update":"2020-06-04T08:49:52.683590Z"},{"marker":"1_1591260592.527939_578.1","last_update":"2020-06-04T08:49:52.527939Z"}] 2020-06-04T08:50:30.128 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-06-04T08:50:30.128 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:30.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:30.130 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:30.172 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:30.173 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:30.448 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7644369918941386260},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260565.934456_240.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:49:25.934456Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591260592.648943_524.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:49:52.648943Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591260592.683590_339.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:49:52.683590Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260592.527939_578.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:49:52.527939Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:50:30.449 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-06-04T08:50:30.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:30.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:30.450 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:30.529 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:30.529 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:30.777 INFO:teuthology.orchestra.run.smithi196.stdout:[{"marker":"1_1591260535.845354_196.1","last_update":"2020-06-04T08:48:55.845354Z"},{"marker":"1_1591260592.586652_503.1","last_update":"2020-06-04T08:49:52.586652Z"},{"marker":"1_1591260592.499879_322.1","last_update":"2020-06-04T08:49:52.499879Z"},{"marker":"1_1591260592.626276_522.1","last_update":"2020-06-04T08:49:52.626276Z"}] 2020-06-04T08:50:30.778 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-06-04T08:50:30.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:30.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:30.780 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:30.832 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:30.833 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:31.081 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1308327179334993417},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260535.845354_196.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:48:55.845354Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591260592.586652_503.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:49:52.586652Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591260592.499879_322.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:49:52.499879Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260592.626276_522.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:49:52.626276Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:50:31.082 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-06-04T08:50:31.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:31.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:31.084 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:31.124 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:31.124 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:31.354 INFO:teuthology.orchestra.run.smithi196.stdout:[{"marker":"1_1591260535.845354_196.1","last_update":"2020-06-04T08:48:55.845354Z"},{"marker":"1_1591260592.586652_503.1","last_update":"2020-06-04T08:49:52.586652Z"},{"marker":"1_1591260592.499879_322.1","last_update":"2020-06-04T08:49:52.499879Z"},{"marker":"1_1591260592.626276_522.1","last_update":"2020-06-04T08:49:52.626276Z"}] 2020-06-04T08:50:31.355 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-06-04T08:50:31.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:31.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:31.356 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:31.398 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:31.399 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:31.648 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":16507416453976119466},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260535.845354_196.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:48:55.845354Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591260592.586652_503.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:49:52.586652Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591260592.499879_322.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:49:52.499879Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260592.626276_522.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:49:52.626276Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:50:31.649 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-06-04T08:50:31.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:31.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:31.652 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:31.730 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:31.731 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:31.969 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260535.818761_168.1","last_update":"2020-06-04T08:48:55.818761Z"},{"marker":"1_1591260592.562474_388.1","last_update":"2020-06-04T08:49:52.562474Z"},{"marker":"1_1591260592.565189_243.1","last_update":"2020-06-04T08:49:52.565189Z"},{"marker":"1_1591260609.281837_314.1","last_update":"2020-06-04T08:50:09.281837Z"}] 2020-06-04T08:50:31.969 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-06-04T08:50:31.970 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:31.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:31.972 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:32.055 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:32.055 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:32.324 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3830535810264008500},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260535.818761_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:48:55.818761Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591260592.562474_388.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:49:52.562474Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591260592.565189_243.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:49:52.565189Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260609.281837_314.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:50:09.281837Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:50:32.325 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-06-04T08:50:32.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:32.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:32.327 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:32.407 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:32.408 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:32.651 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260535.818761_168.1","last_update":"2020-06-04T08:48:55.818761Z"},{"marker":"1_1591260592.562474_388.1","last_update":"2020-06-04T08:49:52.562474Z"},{"marker":"1_1591260592.565189_243.1","last_update":"2020-06-04T08:49:52.565189Z"},{"marker":"1_1591260609.281837_314.1","last_update":"2020-06-04T08:50:09.281837Z"}] 2020-06-04T08:50:32.652 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-06-04T08:50:32.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:32.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:32.653 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:32.698 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:32.698 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:32.931 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9078143951431167771},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260535.818761_168.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:48:55.818761Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591260592.562474_388.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:49:52.562474Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591260592.565189_243.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:49:52.565189Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260609.281837_314.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:50:09.281837Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:50:32.932 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-06-04T08:50:32.933 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2020-06-04T08:50:32.934 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-62 2020-06-04T08:50:32.981 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gpxthc-63 2020-06-04T08:50:33.054 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gpxthc-64 2020-06-04T08:50:33.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:33.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:33.716 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:33.774 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:33.775 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:33.994 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=387a2dc2-99d4-41c4-ad70-50eff9e4eab1 2020-06-04T08:50:34.028 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260633.095182_294.1","last_update":"2020-06-04T08:50:33.095182Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:50:34.029 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:50:34.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:34.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:34.031 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:34.078 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:34.079 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:34.290 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260633.095182_294.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:50:33.095182Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:50:34.291 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:50:34.291 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:50:34.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:34.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:34.293 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:34.337 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:34.337 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:34.574 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260633.095182_294.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:50:33.095182Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:50:34.575 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:50:34.576 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:50:34.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:34.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:34.578 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:34.620 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:34.620 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:34.888 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260633.095182_294.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:50:33.095182Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:50:34.889 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:50:34.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:34.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:34.890 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:34.932 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:34.933 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:35.188 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.42.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.74.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.49.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.52.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:35.189 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-62 2020-06-04T08:50:35.190 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:35.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:35.194 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:35.239 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:35.239 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:35.493 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5","timestamp":"2020-06-04T08:50:33.274164Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.74.5","timestamp":"2020-06-04T08:50:33.319515Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5","timestamp":"2020-06-04T08:50:33.319562Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5","timestamp":"2020-06-04T08:50:33.507392Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:35.493 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-62 2020-06-04T08:50:35.494 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:35.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:35.495 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:35.536 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:35.536 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:35.783 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.42.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.74.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.49.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.52.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:35.783 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-62 2020-06-04T08:50:35.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:35.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:35.785 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:35.868 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:35.869 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:36.125 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5","timestamp":"2020-06-04T08:50:33.309468Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.74.5","timestamp":"2020-06-04T08:50:33.309496Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5","timestamp":"2020-06-04T08:50:33.269991Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5","timestamp":"2020-06-04T08:50:33.545338Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:36.126 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-62 2020-06-04T08:50:36.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:36.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:36.128 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:36.215 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:36.216 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:36.470 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.42.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.74.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.49.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.52.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:36.471 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-62 2020-06-04T08:50:36.471 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:36.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:36.473 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:36.519 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:36.519 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:36.796 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.42.5","timestamp":"2020-06-04T08:50:33.224108Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.74.5","timestamp":"2020-06-04T08:50:33.172447Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.49.5","timestamp":"2020-06-04T08:50:33.119495Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.52.5","timestamp":"2020-06-04T08:50:33.271598Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:36.797 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-62 2020-06-04T08:50:36.798 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:36.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:36.799 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:36.899 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:36.907 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:37.143 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000001.32.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.101.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.60.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.64.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:37.144 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-62 2020-06-04T08:50:37.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:37.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:37.146 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:37.191 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:37.191 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:37.471 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.32.5","timestamp":"2020-06-04T08:50:33.534357Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.101.5","timestamp":"2020-06-04T08:50:33.577470Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5","timestamp":"2020-06-04T08:50:33.577496Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5","timestamp":"2020-06-04T08:50:33.809425Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:37.472 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-62 2020-06-04T08:50:37.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:37.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:37.475 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:37.516 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:37.516 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:37.759 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000001.32.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.101.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.60.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.64.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:37.760 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-62 2020-06-04T08:50:37.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:37.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:37.762 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:37.800 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:37.800 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:38.067 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.32.5","timestamp":"2020-06-04T08:50:33.574245Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.101.5","timestamp":"2020-06-04T08:50:33.574227Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5","timestamp":"2020-06-04T08:50:33.532262Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5","timestamp":"2020-06-04T08:50:33.769811Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:38.067 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-62 2020-06-04T08:50:38.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:38.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:38.069 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:38.149 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:38.150 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:38.403 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.35.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.94.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.40.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.63.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:38.403 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-62 2020-06-04T08:50:38.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:38.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:38.405 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:38.486 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:38.487 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:38.752 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.35.5","timestamp":"2020-06-04T08:50:33.553366Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.94.5","timestamp":"2020-06-04T08:50:33.549354Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.40.5","timestamp":"2020-06-04T08:50:33.508842Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5","timestamp":"2020-06-04T08:50:33.753419Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:38.752 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-62 2020-06-04T08:50:38.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:38.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:38.754 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:38.836 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:38.836 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:39.075 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.35.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.94.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.40.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.63.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:39.076 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-62 2020-06-04T08:50:39.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:39.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:39.078 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:39.129 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:39.129 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:39.371 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.35.5","timestamp":"2020-06-04T08:50:33.551558Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.94.5","timestamp":"2020-06-04T08:50:33.551580Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.40.5","timestamp":"2020-06-04T08:50:33.509760Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5","timestamp":"2020-06-04T08:50:33.720031Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:39.371 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-62 2020-06-04T08:50:39.372 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-62 zones={test-zone2, test-zone1} 2020-06-04T08:50:39.383 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:50:39.390 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:39.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:39.391 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:39.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:39.392 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:50:39.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:39.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:39.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:39.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:39.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:39.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:39.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:39.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:39.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:39.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:39.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:40.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:40.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:40.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:40.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:40.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:40.605 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-62 zones={test-zone2, test-zone1} 2020-06-04T08:50:40.605 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-62 zones={test-zone3, test-zone1} 2020-06-04T08:50:40.615 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:50:40.621 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:40.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:40.622 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:40.623 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:40.623 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:50:40.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:40.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:40.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:40.631 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:40.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:40.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:40.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:40.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:40.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:41.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:41.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:41.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:41.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:41.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:41.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:41.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:41.760 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-62 zones={test-zone3, test-zone1} 2020-06-04T08:50:41.761 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-62 zones={test-zone3, test-zone2} 2020-06-04T08:50:41.772 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:50:41.778 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:41.779 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:41.780 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:41.781 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:41.781 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:50:41.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:41.788 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:41.788 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:41.789 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:41.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:41.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:41.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:42.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:42.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:42.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:42.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:42.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:42.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:42.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:42.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:42.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:43.017 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-62 zones={test-zone3, test-zone2} 2020-06-04T08:50:43.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:43.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:43.023 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:43.075 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:43.075 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:43.326 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.54.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.56.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.54.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.56.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:43.326 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-63 2020-06-04T08:50:43.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:43.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:43.328 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:43.373 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:43.374 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:43.627 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5","timestamp":"2020-06-04T08:50:33.763458Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-06-04T08:50:33.923566Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5","timestamp":"2020-06-04T08:50:33.882002Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-06-04T08:50:33.923552Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:43.627 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-63 2020-06-04T08:50:43.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:43.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:43.629 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:43.673 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:43.673 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:43.926 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.54.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.56.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.54.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.56.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:43.927 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-63 2020-06-04T08:50:43.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:43.929 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:43.929 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:44.013 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:44.015 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:44.292 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5","timestamp":"2020-06-04T08:50:33.705369Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-06-04T08:50:33.957531Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.54.5","timestamp":"2020-06-04T08:50:33.871540Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-06-04T08:50:33.871918Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:44.293 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-63 2020-06-04T08:50:44.294 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:44.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:44.295 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:44.337 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:44.337 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:44.599 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.54.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.56.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.54.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.56.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:44.600 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-63 2020-06-04T08:50:44.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:44.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:44.602 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:44.648 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:44.648 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:44.915 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.54.5","timestamp":"2020-06-04T08:50:33.426698Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-06-04T08:50:33.377535Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.54.5","timestamp":"2020-06-04T08:50:33.330184Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-06-04T08:50:33.475478Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:44.915 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-63 2020-06-04T08:50:44.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:44.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:44.919 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:44.956 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:44.957 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:45.186 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000001.62.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.72.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.65.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.67.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:45.186 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-63 2020-06-04T08:50:45.187 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:45.187 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:45.188 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:45.234 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:45.234 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:45.511 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-04T08:50:33.536106Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.72.5","timestamp":"2020-06-04T08:50:33.536300Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.65.5","timestamp":"2020-06-04T08:50:33.536515Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5","timestamp":"2020-06-04T08:50:33.577507Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:45.512 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-63 2020-06-04T08:50:45.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:45.513 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:45.514 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:45.555 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:45.555 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:45.817 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000001.62.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.72.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.65.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.67.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:45.824 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-63 2020-06-04T08:50:45.825 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:45.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:45.826 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:45.872 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:45.872 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:46.142 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-04T08:50:33.536398Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.72.5","timestamp":"2020-06-04T08:50:33.569436Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.65.5","timestamp":"2020-06-04T08:50:33.536500Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5","timestamp":"2020-06-04T08:50:33.536482Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:46.143 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-63 2020-06-04T08:50:46.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:46.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:46.145 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:46.226 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:46.226 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:46.469 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.62.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.61.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.50.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.52.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:46.470 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-63 2020-06-04T08:50:46.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:46.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:46.472 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:46.555 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:46.556 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:46.817 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-04T08:50:33.749362Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5","timestamp":"2020-06-04T08:50:33.749381Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5","timestamp":"2020-06-04T08:50:33.753447Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5","timestamp":"2020-06-04T08:50:33.709056Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:46.818 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-63 2020-06-04T08:50:46.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:46.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:46.820 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:46.899 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:46.899 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:47.128 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.62.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.61.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.50.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.52.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:47.129 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-63 2020-06-04T08:50:47.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:47.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:47.132 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:47.180 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:47.181 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:47.436 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-04T08:50:33.722736Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5","timestamp":"2020-06-04T08:50:33.749287Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5","timestamp":"2020-06-04T08:50:33.722854Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5","timestamp":"2020-06-04T08:50:33.722832Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:47.437 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-63 2020-06-04T08:50:47.438 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-63 zones={test-zone2, test-zone1} 2020-06-04T08:50:47.451 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:50:47.458 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:47.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:47.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:47.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:47.460 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:50:47.468 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:47.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:47.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:47.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:47.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:47.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:47.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:47.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:47.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:47.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:48.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:48.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:48.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:48.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:48.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:48.516 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:48.623 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-63 zones={test-zone2, test-zone1} 2020-06-04T08:50:48.624 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-63 zones={test-zone3, test-zone1} 2020-06-04T08:50:48.634 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:50:48.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:48.641 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:48.642 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:48.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:48.643 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:50:48.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:48.650 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:48.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:48.651 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:48.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:48.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:48.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:48.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:49.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:49.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:49.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:49.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:49.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:49.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:49.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:49.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:49.796 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-63 zones={test-zone3, test-zone1} 2020-06-04T08:50:49.797 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-63 zones={test-zone3, test-zone2} 2020-06-04T08:50:49.815 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:50:49.828 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:49.829 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:49.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:49.830 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:49.830 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:50:49.838 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:49.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:49.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:49.839 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:49.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:49.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:50.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:50.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:50.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:50.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:50.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:50.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:50.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:50.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:50.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:50.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:51.017 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-63 zones={test-zone3, test-zone2} 2020-06-04T08:50:51.018 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:51.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:51.019 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:51.060 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:51.061 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:51.332 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.72.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.74.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.97.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.47.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:51.332 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-64 2020-06-04T08:50:51.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:51.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:51.334 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:51.380 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:51.381 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:51.645 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.72.5","timestamp":"2020-06-04T08:50:33.927412Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.74.5","timestamp":"2020-06-04T08:50:34.159460Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.97.5","timestamp":"2020-06-04T08:50:34.163439Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5","timestamp":"2020-06-04T08:50:33.923524Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:51.649 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-64 2020-06-04T08:50:51.650 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:51.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:51.651 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:51.695 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:51.695 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:51.931 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.72.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.74.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.97.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.47.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:51.932 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-64 2020-06-04T08:50:51.932 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:51.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:51.934 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:52.016 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:52.016 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:52.283 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.72.5","timestamp":"2020-06-04T08:50:33.917364Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.74.5","timestamp":"2020-06-04T08:50:34.153710Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.97.5","timestamp":"2020-06-04T08:50:34.105325Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5","timestamp":"2020-06-04T08:50:33.909350Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:52.284 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-64 2020-06-04T08:50:52.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:52.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:52.285 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:52.368 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:52.368 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:52.625 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.72.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.74.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.97.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.47.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:52.626 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-64 2020-06-04T08:50:52.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:52.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:52.627 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:52.671 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:52.672 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:52.941 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.72.5","timestamp":"2020-06-04T08:50:33.615622Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.74.5","timestamp":"2020-06-04T08:50:33.567074Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.97.5","timestamp":"2020-06-04T08:50:33.519220Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.47.5","timestamp":"2020-06-04T08:50:33.668725Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:52.942 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-64 2020-06-04T08:50:52.943 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:52.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:52.945 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:53.022 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:53.022 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:53.263 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000001.70.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.72.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.89.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.51.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:53.264 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-64 2020-06-04T08:50:53.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:53.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:53.266 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:53.315 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:53.315 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:53.619 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.70.5","timestamp":"2020-06-04T08:50:33.973455Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.72.5","timestamp":"2020-06-04T08:50:34.213518Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.89.5","timestamp":"2020-06-04T08:50:33.973431Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5","timestamp":"2020-06-04T08:50:33.931188Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:53.619 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-64 2020-06-04T08:50:53.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:53.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:53.621 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:53.661 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:53.662 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:53.905 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000001.70.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.72.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.89.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.51.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:53.906 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-64 2020-06-04T08:50:53.906 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:53.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:53.907 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:53.950 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:53.951 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:54.219 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.70.5","timestamp":"2020-06-04T08:50:33.977486Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.72.5","timestamp":"2020-06-04T08:50:34.209450Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.89.5","timestamp":"2020-06-04T08:50:33.935737Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5","timestamp":"2020-06-04T08:50:33.977463Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:54.220 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-64 2020-06-04T08:50:54.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:54.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:54.221 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:54.302 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:54.303 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:54.558 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.64.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.62.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.69.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.42.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:54.558 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-64 2020-06-04T08:50:54.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:54.559 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:54.560 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:54.652 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:54.652 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:54.900 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5","timestamp":"2020-06-04T08:50:33.715110Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-04T08:50:33.745350Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.69.5","timestamp":"2020-06-04T08:50:33.715371Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5","timestamp":"2020-06-04T08:50:33.715128Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:54.901 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-64 2020-06-04T08:50:54.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:54.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:54.902 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:54.980 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:54.980 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:55.234 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000001.64.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.62.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.69.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.42.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:50:55.235 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-64 2020-06-04T08:50:55.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:55.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:50:55.236 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:50:55.281 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:55.281 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:55.539 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5","timestamp":"2020-06-04T08:50:33.724733Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-04T08:50:33.791585Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.69.5","timestamp":"2020-06-04T08:50:33.759553Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5","timestamp":"2020-06-04T08:50:33.759499Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:50:55.540 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-64 2020-06-04T08:50:55.541 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-64 zones={test-zone2, test-zone1} 2020-06-04T08:50:55.552 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:50:55.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:55.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:55.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:55.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:55.561 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:50:55.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:55.570 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:55.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:55.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:55.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:55.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:55.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:55.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:55.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:56.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:56.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:56.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:56.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:56.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:56.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:56.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:56.742 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-64 zones={test-zone2, test-zone1} 2020-06-04T08:50:56.743 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-64 zones={test-zone3, test-zone1} 2020-06-04T08:50:56.752 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:50:56.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:56.759 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:56.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:56.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:56.761 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:50:56.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:56.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:56.768 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:56.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:56.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:56.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:56.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:57.064 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:57.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:57.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:57.353 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:57.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:57.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:57.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:57.705 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:57.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:57.944 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-64 zones={test-zone3, test-zone1} 2020-06-04T08:50:57.944 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-64 zones={test-zone3, test-zone2} 2020-06-04T08:50:57.955 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:50:57.962 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:57.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:57.963 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:57.964 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:57.964 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:50:57.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:50:57.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:50:57.974 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:50:57.974 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:50:57.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:58.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:58.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:50:58.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:58.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:58.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:50:58.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:58.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:58.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:50:58.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:58.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:58.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:50:59.109 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-64 zones={test-zone3, test-zone2} 2020-06-04T08:50:59.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'gpxthc-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:50:59.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'gpxthc-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:50:59.110 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket gpxthc-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:50:59.158 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:59.158 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:59.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'gpxthc-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:50:59.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'gpxthc-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:50:59.485 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket gpxthc-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:50:59.564 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:59.565 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:50:59.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'gpxthc-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:50:59.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'gpxthc-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:50:59.904 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket gpxthc-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:50:59.989 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:50:59.990 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:00.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:00.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:00.332 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:00.412 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:00.412 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:00.684 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=387a2dc2-99d4-41c4-ad70-50eff9e4eab1 2020-06-04T08:51:00.710 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260660.272240_306.1","last_update":"2020-06-04T08:51:00.272240Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:51:00.711 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:51:00.711 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:00.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:00.713 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:00.757 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:00.758 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:00.967 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260660.272240_306.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:51:00.272240Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:51:00.967 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:51:00.968 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:51:00.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:00.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:00.971 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:01.011 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:01.011 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:01.265 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260660.272240_306.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:51:00.272240Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:51:01.266 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:51:01.266 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:51:01.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:01.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:01.268 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:01.310 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:01.311 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:01.582 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260660.272240_306.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:51:00.272240Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:51:01.582 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:51:02.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'gpxthc-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:51:02.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'gpxthc-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:51:02.207 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket gpxthc-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:51:02.249 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:02.249 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:02.589 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'gpxthc-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:51:02.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'gpxthc-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:51:02.591 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket gpxthc-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:51:02.675 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:02.675 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:03.008 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'gpxthc-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:51:03.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'gpxthc-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:51:03.009 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket gpxthc-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:51:03.088 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:03.088 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:03.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:03.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:03.429 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:03.508 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:03.508 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:03.756 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000006.51.1"},{"key":1,"val":"00000000004.65.1"},{"key":2,"val":"00000000006.80.1"},{"key":3,"val":"00000000004.52.1"},{"key":4,"val":"00000000006.55.1"},{"key":5,"val":"00000000004.40.1"},{"key":6,"val":"00000000004.36.1"},{"key":7,"val":"00000000006.64.1"},{"key":8,"val":"00000000004.73.1"},{"key":9,"val":"00000000004.200.1"},{"key":10,"val":"00000000004.108.1"}]} 2020-06-04T08:51:03.756 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-62 2020-06-04T08:51:03.757 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:03.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:03.758 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:03.803 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:03.803 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:04.060 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:51:04.061 WARNING:rgw_multi.tests:bucket gpxthc-62 zone test-zone2 behind zone test-zone1: shard=0 master=00000000006.51.1 target=, shard=1 master=00000000004.65.1 target=, shard=2 master=00000000006.80.1 target=, shard=3 master=00000000004.52.1 target=, shard=4 master=00000000006.55.1 target=, shard=5 master=00000000004.40.1 target=, shard=6 master=00000000004.36.1 target=, shard=7 master=00000000006.64.1 target=, shard=8 master=00000000004.73.1 target=, shard=9 master=00000000004.200.1 target=, shard=10 master=00000000004.108.1 target= 2020-06-04T08:51:09.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:09.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:09.064 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:09.123 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:09.123 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:09.384 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:51:09.385 WARNING:rgw_multi.tests:bucket gpxthc-62 zone test-zone2 behind zone test-zone1: shard=0 master=00000000006.51.1 target=, shard=1 master=00000000004.65.1 target=, shard=2 master=00000000006.80.1 target=, shard=3 master=00000000004.52.1 target=, shard=4 master=00000000006.55.1 target=, shard=5 master=00000000004.40.1 target=, shard=6 master=00000000004.36.1 target=, shard=7 master=00000000006.64.1 target=, shard=8 master=00000000004.73.1 target=, shard=9 master=00000000004.200.1 target=, shard=10 master=00000000004.108.1 target= 2020-06-04T08:51:14.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:14.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:14.388 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:14.432 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:14.433 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:14.697 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.51.1","timestamp":"2020-06-04T08:51:09.545117Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.65.1","timestamp":"2020-06-04T08:51:12.499597Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.80.1","timestamp":"2020-06-04T08:51:09.572635Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.52.1","timestamp":"2020-06-04T08:51:09.534830Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.55.1","timestamp":"2020-06-04T08:51:09.548760Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.40.1","timestamp":"2020-06-04T08:51:12.453357Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.36.1","timestamp":"2020-06-04T08:51:09.545166Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.64.1","timestamp":"2020-06-04T08:51:09.591465Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.73.1","timestamp":"2020-06-04T08:51:09.545156Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.200.1","timestamp":"2020-06-04T08:51:12.453378Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.108.1","timestamp":"2020-06-04T08:51:09.544105Z"}}}] 2020-06-04T08:51:14.698 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-62 2020-06-04T08:51:14.699 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:14.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:14.700 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:14.746 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:14.746 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:14.991 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000006.51.1"},{"key":1,"val":"00000000004.65.1"},{"key":2,"val":"00000000006.80.1"},{"key":3,"val":"00000000004.52.1"},{"key":4,"val":"00000000006.55.1"},{"key":5,"val":"00000000004.40.1"},{"key":6,"val":"00000000004.36.1"},{"key":7,"val":"00000000006.64.1"},{"key":8,"val":"00000000004.73.1"},{"key":9,"val":"00000000004.200.1"},{"key":10,"val":"00000000004.108.1"}]} 2020-06-04T08:51:14.992 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-62 2020-06-04T08:51:14.992 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:14.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:14.994 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:15.081 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:15.082 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:15.350 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.51.1","timestamp":"2020-06-04T08:51:09.526805Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.65.1","timestamp":"2020-06-04T08:51:12.477393Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.80.1","timestamp":"2020-06-04T08:51:09.514479Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.52.1","timestamp":"2020-06-04T08:51:09.521914Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.55.1","timestamp":"2020-06-04T08:51:09.521932Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.40.1","timestamp":"2020-06-04T08:51:12.475850Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.36.1","timestamp":"2020-06-04T08:51:09.521979Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.64.1","timestamp":"2020-06-04T08:51:09.521898Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.73.1","timestamp":"2020-06-04T08:51:09.521948Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.200.1","timestamp":"2020-06-04T08:51:12.476816Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.108.1","timestamp":"2020-06-04T08:51:09.517180Z"}}}] 2020-06-04T08:51:15.351 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-62 2020-06-04T08:51:15.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:15.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:15.354 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:15.433 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:15.433 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:15.685 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000006.51.1"},{"key":1,"val":"00000000004.65.1"},{"key":2,"val":"00000000006.80.1"},{"key":3,"val":"00000000004.52.1"},{"key":4,"val":"00000000006.55.1"},{"key":5,"val":"00000000004.40.1"},{"key":6,"val":"00000000004.36.1"},{"key":7,"val":"00000000006.64.1"},{"key":8,"val":"00000000004.73.1"},{"key":9,"val":"00000000004.200.1"},{"key":10,"val":"00000000004.108.1"}]} 2020-06-04T08:51:15.686 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-62 2020-06-04T08:51:15.686 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:15.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:15.687 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:15.732 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:15.733 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:15.997 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.51.1","timestamp":"2020-06-04T08:51:02.535609Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.65.1","timestamp":"2020-06-04T08:51:02.536234Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.80.1","timestamp":"2020-06-04T08:51:02.535746Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.52.1","timestamp":"2020-06-04T08:51:02.538294Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.55.1","timestamp":"2020-06-04T08:51:02.536202Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.40.1","timestamp":"2020-06-04T08:51:02.536576Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.36.1","timestamp":"2020-06-04T08:51:02.537334Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.64.1","timestamp":"2020-06-04T08:51:02.539427Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.73.1","timestamp":"2020-06-04T08:51:02.538028Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.200.1","timestamp":"2020-06-04T08:51:02.538356Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.108.1","timestamp":"2020-06-04T08:51:02.535570Z"}}}] 2020-06-04T08:51:15.998 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-62 2020-06-04T08:51:15.998 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:15.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:15.999 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:16.073 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:16.073 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:16.301 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000004.42.5"},{"key":1,"val":"00000000002.73.1"},{"key":2,"val":"00000000004.107.5"},{"key":3,"val":"00000000002.38.1"},{"key":4,"val":"00000000004.66.5"},{"key":5,"val":"00000000002.35.1"},{"key":6,"val":"00000000002.56.1"},{"key":7,"val":"00000000004.80.5"},{"key":8,"val":"00000000002.64.1"},{"key":9,"val":"00000000002.192.1"},{"key":10,"val":"00000000002.95.1"}]} 2020-06-04T08:51:16.302 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-62 2020-06-04T08:51:16.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:16.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:16.303 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:16.349 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:16.349 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:16.624 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000003.35.1","timestamp":"2020-06-04T08:51:09.201404Z"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000003.103.1","timestamp":"2020-06-04T08:51:09.205759Z"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000003.62.1","timestamp":"2020-06-04T08:51:09.205662Z"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000003.75.1","timestamp":"2020-06-04T08:51:09.205744Z"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:51:16.625 WARNING:rgw_multi.tests:bucket gpxthc-62 zone test-zone1 behind zone test-zone2: shard=0 master=00000000004.42.5 target=00000000003.35.1, shard=1 master=00000000002.73.1 target=, shard=2 master=00000000004.107.5 target=00000000003.103.1, shard=3 master=00000000002.38.1 target=, shard=4 master=00000000004.66.5 target=00000000003.62.1, shard=5 master=00000000002.35.1 target=, shard=6 master=00000000002.56.1 target=, shard=7 master=00000000004.80.5 target=00000000003.75.1, shard=8 master=00000000002.64.1 target=, shard=9 master=00000000002.192.1 target=, shard=10 master=00000000002.95.1 target= 2020-06-04T08:51:21.627 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:21.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:21.628 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:21.674 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:21.675 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:21.930 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.42.5","timestamp":"2020-06-04T08:51:01.691473Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.73.1","timestamp":"2020-06-04T08:51:20.648657Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.107.5","timestamp":"2020-06-04T08:51:01.639584Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.38.1","timestamp":"2020-06-04T08:51:20.681539Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.66.5","timestamp":"2020-06-04T08:51:01.590245Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.35.1","timestamp":"2020-06-04T08:51:20.648900Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.56.1","timestamp":"2020-06-04T08:51:20.641888Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.80.5","timestamp":"2020-06-04T08:51:01.738277Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.64.1","timestamp":"2020-06-04T08:51:20.632437Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.192.1","timestamp":"2020-06-04T08:51:20.635301Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.95.1","timestamp":"2020-06-04T08:51:20.681552Z"}}}] 2020-06-04T08:51:21.931 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-62 2020-06-04T08:51:21.931 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:21.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:21.933 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:21.973 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:21.974 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:22.205 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000004.42.5"},{"key":1,"val":"00000000002.73.1"},{"key":2,"val":"00000000004.107.5"},{"key":3,"val":"00000000002.38.1"},{"key":4,"val":"00000000004.66.5"},{"key":5,"val":"00000000002.35.1"},{"key":6,"val":"00000000002.56.1"},{"key":7,"val":"00000000004.80.5"},{"key":8,"val":"00000000002.64.1"},{"key":9,"val":"00000000002.192.1"},{"key":10,"val":"00000000002.95.1"}]} 2020-06-04T08:51:22.206 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-62 2020-06-04T08:51:22.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:22.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:22.208 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:22.257 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:22.257 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:22.528 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.42.5","timestamp":"2020-06-04T08:51:01.691473Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.73.1","timestamp":"2020-06-04T08:51:20.650495Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.107.5","timestamp":"2020-06-04T08:51:01.639584Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.38.1","timestamp":"2020-06-04T08:51:20.702195Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.66.5","timestamp":"2020-06-04T08:51:01.590245Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.35.1","timestamp":"2020-06-04T08:51:20.690104Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.56.1","timestamp":"2020-06-04T08:51:20.681825Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.80.5","timestamp":"2020-06-04T08:51:01.738277Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.64.1","timestamp":"2020-06-04T08:51:20.650429Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.192.1","timestamp":"2020-06-04T08:51:20.702206Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.95.1","timestamp":"2020-06-04T08:51:20.702173Z"}}}] 2020-06-04T08:51:22.529 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-62 2020-06-04T08:51:22.529 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:22.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:22.531 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:22.573 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:22.574 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:22.835 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000004.45.5"},{"key":1,"val":"00000000002.60.1"},{"key":2,"val":"00000000004.100.5"},{"key":3,"val":"00000000002.41.1"},{"key":4,"val":"00000000004.46.5"},{"key":5,"val":"00000000002.31.1"},{"key":6,"val":"00000000002.36.1"},{"key":7,"val":"00000000004.79.5"},{"key":8,"val":"00000000002.66.1"},{"key":9,"val":"00000000002.204.1"},{"key":10,"val":"00000000002.112.1"}]} 2020-06-04T08:51:22.835 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-62 2020-06-04T08:51:22.836 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:22.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:22.837 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:22.918 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:22.919 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:23.196 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.45.5","timestamp":"2020-06-04T08:51:01.691473Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.60.1","timestamp":"2020-06-04T08:51:20.648579Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.100.5","timestamp":"2020-06-04T08:51:01.639584Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.41.1","timestamp":"2020-06-04T08:51:20.635485Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.46.5","timestamp":"2020-06-04T08:51:01.590245Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.31.1","timestamp":"2020-06-04T08:51:20.665420Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.36.1","timestamp":"2020-06-04T08:51:20.677733Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.79.5","timestamp":"2020-06-04T08:51:01.738277Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.66.1","timestamp":"2020-06-04T08:51:20.642723Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.204.1","timestamp":"2020-06-04T08:51:20.665388Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.112.1","timestamp":"2020-06-04T08:51:20.635520Z"}}}] 2020-06-04T08:51:23.197 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-62 2020-06-04T08:51:23.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:23.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:23.198 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:23.281 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:23.281 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:23.540 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000004.45.5"},{"key":1,"val":"00000000002.60.1"},{"key":2,"val":"00000000004.100.5"},{"key":3,"val":"00000000002.41.1"},{"key":4,"val":"00000000004.46.5"},{"key":5,"val":"00000000002.31.1"},{"key":6,"val":"00000000002.36.1"},{"key":7,"val":"00000000004.79.5"},{"key":8,"val":"00000000002.66.1"},{"key":9,"val":"00000000002.204.1"},{"key":10,"val":"00000000002.112.1"}]} 2020-06-04T08:51:23.541 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-62 2020-06-04T08:51:23.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:23.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:23.542 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:23.586 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:23.587 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:23.846 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.45.5","timestamp":"2020-06-04T08:51:01.691473Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.60.1","timestamp":"2020-06-04T08:51:20.567926Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.100.5","timestamp":"2020-06-04T08:51:01.639584Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.41.1","timestamp":"2020-06-04T08:51:20.565852Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.46.5","timestamp":"2020-06-04T08:51:01.590245Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.31.1","timestamp":"2020-06-04T08:51:20.559494Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.36.1","timestamp":"2020-06-04T08:51:20.546082Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.79.5","timestamp":"2020-06-04T08:51:01.738277Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.66.1","timestamp":"2020-06-04T08:51:20.559366Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.204.1","timestamp":"2020-06-04T08:51:20.599572Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.112.1","timestamp":"2020-06-04T08:51:20.546050Z"}}}] 2020-06-04T08:51:23.846 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-62 2020-06-04T08:51:23.847 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-62 zones={test-zone2, test-zone1} 2020-06-04T08:51:23.858 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:51:23.865 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:51:23.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:51:23.866 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:51:23.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:51:23.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:51:23.867 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:51:23.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:51:23.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:51:23.869 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:51:23.879 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:51:23.880 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:51:23.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:51:23.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:51:23.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:51:23.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:51:23.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:51:23.885 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:51:23.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:23.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:24.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:24.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:24.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:24.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:24.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:24.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:24.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:24.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:24.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:24.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:25.035 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:25.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:25.213 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:25.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:25.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:25.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:25.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:25.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:25.765 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:25.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:25.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:26.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:26.156 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-62 zones={test-zone2, test-zone1} 2020-06-04T08:51:26.156 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-62 zones={test-zone3, test-zone1} 2020-06-04T08:51:26.165 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:51:26.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:51:26.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:51:26.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:51:26.174 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:51:26.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:51:26.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:51:26.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:51:26.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:51:26.177 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:51:26.188 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:51:26.189 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:51:26.190 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:51:26.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:51:26.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:51:26.191 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:51:26.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:51:26.192 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:51:26.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:26.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:26.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:26.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:26.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:26.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:26.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:26.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:26.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:27.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:27.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:27.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:27.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:27.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:27.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:27.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:27.692 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:27.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:27.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:27.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:28.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:28.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:28.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:28.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:28.487 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-62 zones={test-zone3, test-zone1} 2020-06-04T08:51:28.488 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-62 zones={test-zone3, test-zone2} 2020-06-04T08:51:28.500 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:51:28.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:51:28.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:51:28.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:51:28.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:51:28.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:51:28.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:51:28.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:51:28.511 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:51:28.516 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:51:28.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:51:28.525 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:51:28.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:51:28.526 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:51:28.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:51:28.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:51:28.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:51:28.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:51:28.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:28.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:28.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:28.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:28.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:28.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:29.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:29.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:29.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:29.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:29.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:29.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:29.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:29.727 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:29.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:29.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:30.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:30.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:30.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:30.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:30.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:30.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:30.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:30.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:30.817 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-62 zones={test-zone3, test-zone2} 2020-06-04T08:51:30.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:30.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:30.820 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:30.863 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:30.863 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:31.107 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000006.70.5"},{"key":1,"val":"00000000004.54.1"},{"key":2,"val":"00000000006.62.5"},{"key":3,"val":"00000000004.103.1"},{"key":4,"val":"00000000006.70.5"},{"key":5,"val":"00000000004.68.1"},{"key":6,"val":"00000000004.82.1"},{"key":7,"val":"00000000006.72.5"},{"key":8,"val":"00000000004.67.1"},{"key":9,"val":"00000000004.44.1"},{"key":10,"val":"00000000004.53.1"}]} 2020-06-04T08:51:31.107 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-63 2020-06-04T08:51:31.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:31.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:31.109 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:31.147 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:31.148 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:31.409 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.70.5","timestamp":"2020-06-04T08:51:09.550115Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1","timestamp":"2020-06-04T08:51:12.451774Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.62.5","timestamp":"2020-06-04T08:51:09.541215Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.103.1","timestamp":"2020-06-04T08:51:09.550135Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.70.5","timestamp":"2020-06-04T08:51:09.571086Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1","timestamp":"2020-06-04T08:51:12.453372Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.82.1","timestamp":"2020-06-04T08:51:09.547485Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.72.5","timestamp":"2020-06-04T08:51:09.550066Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1","timestamp":"2020-06-04T08:51:09.544049Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.44.1","timestamp":"2020-06-04T08:51:12.450227Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1","timestamp":"2020-06-04T08:51:09.572340Z"}}}] 2020-06-04T08:51:31.410 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-63 2020-06-04T08:51:31.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:31.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:31.411 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:31.458 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:31.458 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:31.692 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000006.70.5"},{"key":1,"val":"00000000004.54.1"},{"key":2,"val":"00000000006.62.5"},{"key":3,"val":"00000000004.103.1"},{"key":4,"val":"00000000006.70.5"},{"key":5,"val":"00000000004.68.1"},{"key":6,"val":"00000000004.82.1"},{"key":7,"val":"00000000006.72.5"},{"key":8,"val":"00000000004.67.1"},{"key":9,"val":"00000000004.44.1"},{"key":10,"val":"00000000004.53.1"}]} 2020-06-04T08:51:31.693 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-63 2020-06-04T08:51:31.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:31.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:31.695 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:31.737 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:31.737 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:31.995 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.70.5","timestamp":"2020-06-04T08:51:09.521874Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1","timestamp":"2020-06-04T08:51:12.475932Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.62.5","timestamp":"2020-06-04T08:51:09.519044Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.103.1","timestamp":"2020-06-04T08:51:09.562121Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.70.5","timestamp":"2020-06-04T08:51:09.514439Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1","timestamp":"2020-06-04T08:51:12.476830Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.82.1","timestamp":"2020-06-04T08:51:09.514462Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.72.5","timestamp":"2020-06-04T08:51:09.562143Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1","timestamp":"2020-06-04T08:51:09.514225Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.44.1","timestamp":"2020-06-04T08:51:12.477280Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1","timestamp":"2020-06-04T08:51:09.521827Z"}}}] 2020-06-04T08:51:31.996 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-63 2020-06-04T08:51:31.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:31.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:31.997 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:32.076 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:32.076 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:32.337 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000006.70.5"},{"key":1,"val":"00000000004.54.1"},{"key":2,"val":"00000000006.62.5"},{"key":3,"val":"00000000004.103.1"},{"key":4,"val":"00000000006.70.5"},{"key":5,"val":"00000000004.68.1"},{"key":6,"val":"00000000004.82.1"},{"key":7,"val":"00000000006.72.5"},{"key":8,"val":"00000000004.67.1"},{"key":9,"val":"00000000004.44.1"},{"key":10,"val":"00000000004.53.1"}]} 2020-06-04T08:51:32.338 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-63 2020-06-04T08:51:32.339 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:32.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:32.341 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:32.387 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:32.387 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:32.654 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.70.5","timestamp":"2020-06-04T08:51:01.914463Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1","timestamp":"2020-06-04T08:51:02.955432Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.62.5","timestamp":"2020-06-04T08:51:01.848159Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.103.1","timestamp":"2020-06-04T08:51:02.957517Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.70.5","timestamp":"2020-06-04T08:51:01.794084Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1","timestamp":"2020-06-04T08:51:02.957505Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.82.1","timestamp":"2020-06-04T08:51:02.959950Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.72.5","timestamp":"2020-06-04T08:51:01.962940Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.67.1","timestamp":"2020-06-04T08:51:02.955279Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.44.1","timestamp":"2020-06-04T08:51:02.955967Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1","timestamp":"2020-06-04T08:51:02.960332Z"}}}] 2020-06-04T08:51:32.655 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-63 2020-06-04T08:51:32.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:32.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:32.657 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:32.699 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:32.700 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:32.932 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000004.73.1"},{"key":1,"val":"00000000002.49.1"},{"key":2,"val":"00000000004.77.1"},{"key":3,"val":"00000000002.95.1"},{"key":4,"val":"00000000004.79.1"},{"key":5,"val":"00000000002.74.1"},{"key":6,"val":"00000000002.81.1"},{"key":7,"val":"00000000004.80.1"},{"key":8,"val":"00000000002.71.1"},{"key":9,"val":"00000000002.51.1"},{"key":10,"val":"00000000002.44.1"}]} 2020-06-04T08:51:32.933 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-63 2020-06-04T08:51:32.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:32.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:32.934 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:32.979 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:32.980 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:33.259 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000004.73.1","timestamp":"2020-06-04T08:51:09.205775Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1","timestamp":"2020-06-04T08:51:20.644703Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000004.77.1","timestamp":"2020-06-04T08:51:09.205729Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.95.1","timestamp":"2020-06-04T08:51:20.649125Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000004.79.1","timestamp":"2020-06-04T08:51:09.205791Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.74.1","timestamp":"2020-06-04T08:51:20.678011Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.81.1","timestamp":"2020-06-04T08:51:20.689677Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000004.80.1","timestamp":"2020-06-04T08:51:09.205712Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1","timestamp":"2020-06-04T08:51:20.632399Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1","timestamp":"2020-06-04T08:51:20.689671Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.44.1","timestamp":"2020-06-04T08:51:20.648601Z"}}}] 2020-06-04T08:51:33.260 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-63 2020-06-04T08:51:33.260 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:33.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:33.262 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:33.305 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:33.306 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:33.537 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000004.73.1"},{"key":1,"val":"00000000002.49.1"},{"key":2,"val":"00000000004.77.1"},{"key":3,"val":"00000000002.95.1"},{"key":4,"val":"00000000004.79.1"},{"key":5,"val":"00000000002.74.1"},{"key":6,"val":"00000000002.81.1"},{"key":7,"val":"00000000004.80.1"},{"key":8,"val":"00000000002.71.1"},{"key":9,"val":"00000000002.51.1"},{"key":10,"val":"00000000002.44.1"}]} 2020-06-04T08:51:33.537 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-63 2020-06-04T08:51:33.538 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:33.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:33.540 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:33.586 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:33.587 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:33.853 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000004.73.1","timestamp":"2020-06-04T08:51:09.205758Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1","timestamp":"2020-06-04T08:51:20.631917Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000004.77.1","timestamp":"2020-06-04T08:51:09.205789Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.95.1","timestamp":"2020-06-04T08:51:20.702184Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000004.79.1","timestamp":"2020-06-04T08:51:09.205719Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.74.1","timestamp":"2020-06-04T08:51:20.628189Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.81.1","timestamp":"2020-06-04T08:51:20.690134Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000004.80.1","timestamp":"2020-06-04T08:51:09.205773Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1","timestamp":"2020-06-04T08:51:20.650549Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1","timestamp":"2020-06-04T08:51:20.650454Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.44.1","timestamp":"2020-06-04T08:51:20.650474Z"}}}] 2020-06-04T08:51:33.854 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-63 2020-06-04T08:51:33.854 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:33.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:33.856 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:33.937 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:33.937 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:34.186 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000004.77.5"},{"key":1,"val":"00000000002.47.1"},{"key":2,"val":"00000000004.67.5"},{"key":3,"val":"00000000002.77.1"},{"key":4,"val":"00000000004.66.5"},{"key":5,"val":"00000000002.71.1"},{"key":6,"val":"00000000002.77.1"},{"key":7,"val":"00000000004.68.5"},{"key":8,"val":"00000000002.59.1"},{"key":9,"val":"00000000002.37.1"},{"key":10,"val":"00000000002.42.1"}]} 2020-06-04T08:51:34.187 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-63 2020-06-04T08:51:34.188 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:34.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:34.189 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:34.272 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:34.273 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:34.542 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.77.5","timestamp":"2020-06-04T08:51:01.914463Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.47.1","timestamp":"2020-06-04T08:51:20.608067Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.67.5","timestamp":"2020-06-04T08:51:01.848159Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.77.1","timestamp":"2020-06-04T08:51:20.626510Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.66.5","timestamp":"2020-06-04T08:51:01.794084Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1","timestamp":"2020-06-04T08:51:20.642443Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.77.1","timestamp":"2020-06-04T08:51:20.626377Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.68.5","timestamp":"2020-06-04T08:51:01.962940Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.59.1","timestamp":"2020-06-04T08:51:20.626529Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.37.1","timestamp":"2020-06-04T08:51:20.614047Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.42.1","timestamp":"2020-06-04T08:51:20.642427Z"}}}] 2020-06-04T08:51:34.543 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-63 2020-06-04T08:51:34.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:34.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:34.544 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:34.586 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:34.586 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:34.833 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000004.77.5"},{"key":1,"val":"00000000002.47.1"},{"key":2,"val":"00000000004.67.5"},{"key":3,"val":"00000000002.77.1"},{"key":4,"val":"00000000004.66.5"},{"key":5,"val":"00000000002.71.1"},{"key":6,"val":"00000000002.77.1"},{"key":7,"val":"00000000004.68.5"},{"key":8,"val":"00000000002.59.1"},{"key":9,"val":"00000000002.37.1"},{"key":10,"val":"00000000002.42.1"}]} 2020-06-04T08:51:34.834 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-63 2020-06-04T08:51:34.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:34.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:34.836 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:34.881 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:34.881 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:35.135 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.77.5","timestamp":"2020-06-04T08:51:01.914463Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.47.1","timestamp":"2020-06-04T08:51:20.558993Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.67.5","timestamp":"2020-06-04T08:51:01.848159Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.77.1","timestamp":"2020-06-04T08:51:20.559142Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.66.5","timestamp":"2020-06-04T08:51:01.794084Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.1","timestamp":"2020-06-04T08:51:20.559408Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.77.1","timestamp":"2020-06-04T08:51:20.579565Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.68.5","timestamp":"2020-06-04T08:51:01.962940Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.59.1","timestamp":"2020-06-04T08:51:20.553629Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.37.1","timestamp":"2020-06-04T08:51:20.566083Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.42.1","timestamp":"2020-06-04T08:51:20.591537Z"}}}] 2020-06-04T08:51:35.135 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-63 2020-06-04T08:51:35.136 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-63 zones={test-zone2, test-zone1} 2020-06-04T08:51:35.147 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:51:35.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:51:35.153 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:51:35.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:51:35.154 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:51:35.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:51:35.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:51:35.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:51:35.156 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:51:35.157 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:51:35.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:51:35.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:51:35.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:51:35.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:51:35.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:51:35.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:51:35.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:51:35.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:51:35.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:35.248 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:35.361 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:35.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:35.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:35.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:35.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:35.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:35.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:36.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:36.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:36.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:36.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:36.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:36.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:36.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:36.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:36.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:36.868 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:36.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:37.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:37.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:37.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:37.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:37.428 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-63 zones={test-zone2, test-zone1} 2020-06-04T08:51:37.429 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-63 zones={test-zone3, test-zone1} 2020-06-04T08:51:37.438 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:51:37.445 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:51:37.446 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:51:37.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:51:37.447 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:51:37.448 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:51:37.449 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:51:37.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:51:37.450 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:51:37.451 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:51:37.459 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:51:37.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:51:37.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:51:37.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:51:37.462 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:51:37.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:51:37.463 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:51:37.464 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:51:37.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:37.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:37.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:37.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:37.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:37.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:38.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:38.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:38.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:38.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:38.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:38.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:38.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:38.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:38.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:38.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:38.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:39.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:39.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:39.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:39.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:39.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:39.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:39.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:39.767 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-63 zones={test-zone3, test-zone1} 2020-06-04T08:51:39.767 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-63 zones={test-zone3, test-zone2} 2020-06-04T08:51:39.780 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:51:39.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:51:39.788 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:51:39.789 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:51:39.789 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:51:39.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:51:39.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:51:39.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:51:39.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:51:39.792 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:51:39.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:51:39.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:51:39.801 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:51:39.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:51:39.802 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:51:39.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:51:39.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:51:39.803 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:51:39.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:39.894 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:40.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:40.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:40.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:40.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:40.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:40.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:40.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:40.693 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:40.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:40.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:40.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:41.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:41.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:41.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:41.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:41.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:41.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:41.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:41.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:41.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:41.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:42.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:42.178 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-63 zones={test-zone3, test-zone2} 2020-06-04T08:51:42.179 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:42.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:42.182 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:42.227 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:42.227 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:42.492 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000006.90.5"},{"key":1,"val":"00000000004.71.1"},{"key":2,"val":"00000000006.88.5"},{"key":3,"val":"00000000004.68.1"},{"key":4,"val":"00000000006.107.5"},{"key":5,"val":"00000000004.72.1"},{"key":6,"val":"00000000004.64.1"},{"key":7,"val":"00000000006.57.5"},{"key":8,"val":"00000000004.55.1"},{"key":9,"val":"00000000004.69.1"},{"key":10,"val":"00000000004.73.1"}]} 2020-06-04T08:51:42.493 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-64 2020-06-04T08:51:42.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:42.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:42.495 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:42.540 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:42.541 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:42.797 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.90.5","timestamp":"2020-06-04T08:51:09.571521Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1","timestamp":"2020-06-04T08:51:12.495539Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.88.5","timestamp":"2020-06-04T08:51:09.548728Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1","timestamp":"2020-06-04T08:51:09.587416Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.107.5","timestamp":"2020-06-04T08:51:09.547384Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1","timestamp":"2020-06-04T08:51:12.495549Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1","timestamp":"2020-06-04T08:51:09.571504Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.57.5","timestamp":"2020-06-04T08:51:09.535074Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1","timestamp":"2020-06-04T08:51:09.563530Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.1","timestamp":"2020-06-04T08:51:12.495508Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.73.1","timestamp":"2020-06-04T08:51:09.526921Z"}}}] 2020-06-04T08:51:42.798 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-64 2020-06-04T08:51:42.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:42.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:42.799 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:42.836 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:42.837 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:43.089 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000006.90.5"},{"key":1,"val":"00000000004.71.1"},{"key":2,"val":"00000000006.88.5"},{"key":3,"val":"00000000004.68.1"},{"key":4,"val":"00000000006.107.5"},{"key":5,"val":"00000000004.72.1"},{"key":6,"val":"00000000004.64.1"},{"key":7,"val":"00000000006.57.5"},{"key":8,"val":"00000000004.55.1"},{"key":9,"val":"00000000004.69.1"},{"key":10,"val":"00000000004.73.1"}]} 2020-06-04T08:51:43.090 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-64 2020-06-04T08:51:43.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:43.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:43.092 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:43.174 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:43.175 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:43.452 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.90.5","timestamp":"2020-06-04T08:51:09.499447Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1","timestamp":"2020-06-04T08:51:12.477022Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.88.5","timestamp":"2020-06-04T08:51:09.499818Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1","timestamp":"2020-06-04T08:51:09.544757Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.107.5","timestamp":"2020-06-04T08:51:09.544749Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1","timestamp":"2020-06-04T08:51:12.517403Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1","timestamp":"2020-06-04T08:51:09.544737Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.57.5","timestamp":"2020-06-04T08:51:09.544765Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1","timestamp":"2020-06-04T08:51:09.497384Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.1","timestamp":"2020-06-04T08:51:12.475873Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.73.1","timestamp":"2020-06-04T08:51:09.537391Z"}}}] 2020-06-04T08:51:43.452 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-64 2020-06-04T08:51:43.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:43.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:43.454 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:43.534 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:43.534 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:43.794 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000006.90.5"},{"key":1,"val":"00000000004.71.1"},{"key":2,"val":"00000000006.88.5"},{"key":3,"val":"00000000004.68.1"},{"key":4,"val":"00000000006.107.5"},{"key":5,"val":"00000000004.72.1"},{"key":6,"val":"00000000004.64.1"},{"key":7,"val":"00000000006.57.5"},{"key":8,"val":"00000000004.55.1"},{"key":9,"val":"00000000004.69.1"},{"key":10,"val":"00000000004.73.1"}]} 2020-06-04T08:51:43.795 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-64 2020-06-04T08:51:43.796 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:43.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:43.797 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:43.840 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:43.840 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:44.114 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.90.5","timestamp":"2020-06-04T08:51:02.112694Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.71.1","timestamp":"2020-06-04T08:51:03.378182Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.88.5","timestamp":"2020-06-04T08:51:02.061197Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1","timestamp":"2020-06-04T08:51:03.379027Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.107.5","timestamp":"2020-06-04T08:51:02.009609Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1","timestamp":"2020-06-04T08:51:03.382109Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1","timestamp":"2020-06-04T08:51:03.380925Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.57.5","timestamp":"2020-06-04T08:51:02.160695Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1","timestamp":"2020-06-04T08:51:03.379795Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.69.1","timestamp":"2020-06-04T08:51:03.381280Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.73.1","timestamp":"2020-06-04T08:51:03.383751Z"}}}] 2020-06-04T08:51:44.115 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-64 2020-06-04T08:51:44.116 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:44.116 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:44.117 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:44.199 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:44.200 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:44.447 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000004.86.5"},{"key":1,"val":"00000000002.64.1"},{"key":2,"val":"00000000004.88.5"},{"key":3,"val":"00000000002.78.1"},{"key":4,"val":"00000000004.99.5"},{"key":5,"val":"00000000002.70.1"},{"key":6,"val":"00000000002.80.1"},{"key":7,"val":"00000000004.61.5"},{"key":8,"val":"00000000002.57.1"},{"key":9,"val":"00000000002.54.1"},{"key":10,"val":"00000000002.72.1"}]} 2020-06-04T08:51:44.448 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-64 2020-06-04T08:51:44.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:44.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:44.450 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:44.497 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:44.497 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:44.794 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.86.5","timestamp":"2020-06-04T08:51:02.112694Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.64.1","timestamp":"2020-06-04T08:51:20.648832Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.88.5","timestamp":"2020-06-04T08:51:02.061197Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.78.1","timestamp":"2020-06-04T08:51:20.637282Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.99.5","timestamp":"2020-06-04T08:51:02.009609Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.70.1","timestamp":"2020-06-04T08:51:20.689684Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1","timestamp":"2020-06-04T08:51:20.689663Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.61.5","timestamp":"2020-06-04T08:51:02.160695Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.57.1","timestamp":"2020-06-04T08:51:20.689623Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1","timestamp":"2020-06-04T08:51:20.689656Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.72.1","timestamp":"2020-06-04T08:51:20.689647Z"}}}] 2020-06-04T08:51:44.795 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-64 2020-06-04T08:51:44.796 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:44.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:44.799 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:44.847 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:44.847 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:45.329 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":"00000000004.86.5"},{"key":1,"val":"00000000002.64.1"},{"key":2,"val":"00000000004.88.5"},{"key":3,"val":"00000000002.78.1"},{"key":4,"val":"00000000004.99.5"},{"key":5,"val":"00000000002.70.1"},{"key":6,"val":"00000000002.80.1"},{"key":7,"val":"00000000004.61.5"},{"key":8,"val":"00000000002.57.1"},{"key":9,"val":"00000000002.54.1"},{"key":10,"val":"00000000002.72.1"}]} 2020-06-04T08:51:45.330 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-64 2020-06-04T08:51:45.331 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:45.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:45.332 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:45.383 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:45.384 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:45.659 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.86.5","timestamp":"2020-06-04T08:51:02.112694Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.64.1","timestamp":"2020-06-04T08:51:20.673473Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.88.5","timestamp":"2020-06-04T08:51:02.061197Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.78.1","timestamp":"2020-06-04T08:51:20.596698Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.99.5","timestamp":"2020-06-04T08:51:02.009609Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.70.1","timestamp":"2020-06-04T08:51:20.597352Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1","timestamp":"2020-06-04T08:51:20.597339Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.61.5","timestamp":"2020-06-04T08:51:02.160695Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.57.1","timestamp":"2020-06-04T08:51:20.597315Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1","timestamp":"2020-06-04T08:51:20.673505Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.72.1","timestamp":"2020-06-04T08:51:20.646172Z"}}}] 2020-06-04T08:51:45.660 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-64 2020-06-04T08:51:45.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:45.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:45.663 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:45.743 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:45.743 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:46.005 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000004.75.1"},{"key":1,"val":"00000000002.51.1"},{"key":2,"val":"00000000004.76.1"},{"key":3,"val":"00000000002.56.1"},{"key":4,"val":"00000000004.76.1"},{"key":5,"val":"00000000002.57.1"},{"key":6,"val":"00000000002.54.1"},{"key":7,"val":"00000000004.49.1"},{"key":8,"val":"00000000002.50.1"},{"key":9,"val":"00000000002.66.1"},{"key":10,"val":"00000000002.58.1"}]} 2020-06-04T08:51:46.006 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-64 2020-06-04T08:51:46.007 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:46.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:46.020 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:46.097 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:46.098 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:46.377 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000004.75.1","timestamp":"2020-06-04T08:51:09.217406Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1","timestamp":"2020-06-04T08:51:20.551672Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000004.76.1","timestamp":"2020-06-04T08:51:09.213988Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.56.1","timestamp":"2020-06-04T08:51:20.551466Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000004.76.1","timestamp":"2020-06-04T08:51:09.217427Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.57.1","timestamp":"2020-06-04T08:51:20.585691Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1","timestamp":"2020-06-04T08:51:20.551653Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000004.49.1","timestamp":"2020-06-04T08:51:09.213968Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.50.1","timestamp":"2020-06-04T08:51:20.586660Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.66.1","timestamp":"2020-06-04T08:51:20.586682Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.58.1","timestamp":"2020-06-04T08:51:20.605418Z"}}}] 2020-06-04T08:51:46.379 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-64 2020-06-04T08:51:46.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:46.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:46.382 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:46.424 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:46.426 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:46.678 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":"00000000004.75.1"},{"key":1,"val":"00000000002.51.1"},{"key":2,"val":"00000000004.76.1"},{"key":3,"val":"00000000002.56.1"},{"key":4,"val":"00000000004.76.1"},{"key":5,"val":"00000000002.57.1"},{"key":6,"val":"00000000002.54.1"},{"key":7,"val":"00000000004.49.1"},{"key":8,"val":"00000000002.50.1"},{"key":9,"val":"00000000002.66.1"},{"key":10,"val":"00000000002.58.1"}]} 2020-06-04T08:51:46.678 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-64 2020-06-04T08:51:46.679 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:46.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:46.680 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:46.724 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:46.724 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:46.959 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000004.75.1","timestamp":"2020-06-04T08:51:09.219424Z"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.1","timestamp":"2020-06-04T08:51:20.591501Z"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000004.76.1","timestamp":"2020-06-04T08:51:09.215464Z"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.56.1","timestamp":"2020-06-04T08:51:20.591571Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000004.76.1","timestamp":"2020-06-04T08:51:09.219445Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.57.1","timestamp":"2020-06-04T08:51:20.591555Z"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1","timestamp":"2020-06-04T08:51:20.546020Z"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000004.49.1","timestamp":"2020-06-04T08:51:09.219440Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.50.1","timestamp":"2020-06-04T08:51:20.551476Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.66.1","timestamp":"2020-06-04T08:51:20.557889Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.58.1","timestamp":"2020-06-04T08:51:20.543307Z"}}}] 2020-06-04T08:51:46.960 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-64 2020-06-04T08:51:46.962 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-64 zones={test-zone2, test-zone1} 2020-06-04T08:51:46.978 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:51:46.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:51:46.988 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:51:46.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:51:46.989 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:51:46.990 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:51:46.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:51:46.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:51:46.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:51:46.993 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:51:47.003 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:51:47.004 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:51:47.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:51:47.005 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:51:47.007 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:51:47.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:51:47.008 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:51:47.009 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:51:47.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:47.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:47.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:47.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:47.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:47.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:47.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:47.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:47.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:47.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:47.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:48.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:48.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:48.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:48.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:48.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:48.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:48.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:48.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:48.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:48.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:49.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:49.156 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:49.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:49.385 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-64 zones={test-zone2, test-zone1} 2020-06-04T08:51:49.386 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-64 zones={test-zone3, test-zone1} 2020-06-04T08:51:49.405 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:51:49.412 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:51:49.413 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:51:49.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:51:49.414 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:51:49.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:51:49.415 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:51:49.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:51:49.416 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:51:49.417 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:51:49.426 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:51:49.427 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:51:49.427 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:51:49.428 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:51:49.428 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:51:49.429 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:51:49.429 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:51:49.430 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:51:49.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:49.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:49.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:49.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:49.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:49.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:50.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:50.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:50.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:50.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:50.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:50.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:50.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:50.672 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:50.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:50.897 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:50.950 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:51.085 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:51.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:51.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:51.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:51.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:51.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:51.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:51.771 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-64 zones={test-zone3, test-zone1} 2020-06-04T08:51:51.772 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-64 zones={test-zone3, test-zone2} 2020-06-04T08:51:51.784 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:51:51.790 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:51:51.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:51:51.791 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:51:51.792 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:51:51.792 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:51:51.793 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:51:51.793 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:51:51.794 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:51:51.794 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:51:51.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2020-06-04T08:51:51.809 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2020-06-04T08:51:51.810 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2020-06-04T08:51:51.810 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2020-06-04T08:51:51.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2020-06-04T08:51:51.811 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2020-06-04T08:51:51.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2020-06-04T08:51:51.812 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2020-06-04T08:51:51.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:51.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:52.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2020-06-04T08:51:52.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:52.171 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:52.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2020-06-04T08:51:52.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:52.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:52.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2020-06-04T08:51:52.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:52.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:52.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2020-06-04T08:51:53.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:53.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:53.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2020-06-04T08:51:53.310 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:53.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:53.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2020-06-04T08:51:53.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:53.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:53.772 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2020-06-04T08:51:53.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:53.943 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:54.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2020-06-04T08:51:54.172 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-64 zones={test-zone3, test-zone2} 2020-06-04T08:51:54.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:54.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:54.174 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:54.228 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:54.229 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:54.476 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260663.389610_269.3","last_update":"2020-06-04T08:51:03.389610Z"},{"marker":"1_1591260675.256849_596.1","last_update":"2020-06-04T08:51:15.256849Z"},{"marker":"1_1591260675.256849_411.1","last_update":"2020-06-04T08:51:15.256849Z"},{"marker":"1_1591260675.256849_671.2","last_update":"2020-06-04T08:51:15.256849Z"}] 2020-06-04T08:51:54.477 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-06-04T08:51:54.478 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:54.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:54.479 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:54.523 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:54.524 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:54.770 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":11207024744551397021},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260663.389610_269.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:03.389610Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591260675.256849_596.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.256849Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591260675.256849_411.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.256849Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260675.256849_671.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.256849Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:51:54.771 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2020-06-04T08:51:54.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:54.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:54.773 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:54.813 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:54.814 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:55.067 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260663.389610_269.3","last_update":"2020-06-04T08:51:03.389610Z"},{"marker":"1_1591260675.256849_596.1","last_update":"2020-06-04T08:51:15.256849Z"},{"marker":"1_1591260675.256849_411.1","last_update":"2020-06-04T08:51:15.256849Z"},{"marker":"1_1591260675.256849_671.2","last_update":"2020-06-04T08:51:15.256849Z"}] 2020-06-04T08:51:55.067 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-06-04T08:51:55.068 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:55.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:55.069 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:55.152 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:55.152 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:55.415 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":2627143167498148538},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260663.389610_269.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:03.389610Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591260675.256849_596.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.256849Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591260675.256849_411.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.256849Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260675.256849_671.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.256849Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:51:55.416 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2020-06-04T08:51:55.416 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:55.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:55.417 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:55.497 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:55.497 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:55.763 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260663.389610_269.3","last_update":"2020-06-04T08:51:03.389610Z"},{"marker":"1_1591260675.256849_596.1","last_update":"2020-06-04T08:51:15.256849Z"},{"marker":"1_1591260675.256849_411.1","last_update":"2020-06-04T08:51:15.256849Z"},{"marker":"1_1591260675.256849_671.2","last_update":"2020-06-04T08:51:15.256849Z"}] 2020-06-04T08:51:55.764 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-06-04T08:51:55.765 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:55.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:55.767 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:55.812 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:55.813 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:56.085 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7644369918941386260},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260663.389610_269.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:03.389610Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591260675.256849_596.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.256849Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591260675.256849_411.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.256849Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260675.256849_671.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.256849Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:51:56.085 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2020-06-04T08:51:56.086 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:56.086 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:56.088 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:56.163 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:56.164 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:56.418 INFO:teuthology.orchestra.run.smithi196.stdout:[{"marker":"1_1591260675.106698_228.9","last_update":"2020-06-04T08:51:15.106698Z"},{"marker":"1_1591260675.106698_559.9","last_update":"2020-06-04T08:51:15.106698Z"},{"marker":"1_1591260675.106698_374.6","last_update":"2020-06-04T08:51:15.106698Z"},{"marker":"1_1591260675.106698_580.9","last_update":"2020-06-04T08:51:15.106698Z"}] 2020-06-04T08:51:56.419 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-06-04T08:51:56.420 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:56.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:56.422 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:56.464 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:56.465 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:56.711 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":1308327179334993417},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260675.106698_228.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.106698Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591260675.106698_559.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.106698Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591260675.106698_374.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.106698Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260675.106698_580.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.106698Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:51:56.711 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2020-06-04T08:51:56.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:56.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:56.713 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:56.751 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:56.751 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:56.988 INFO:teuthology.orchestra.run.smithi196.stdout:[{"marker":"1_1591260675.106698_228.9","last_update":"2020-06-04T08:51:15.106698Z"},{"marker":"1_1591260675.106698_559.9","last_update":"2020-06-04T08:51:15.106698Z"},{"marker":"1_1591260675.106698_374.6","last_update":"2020-06-04T08:51:15.106698Z"},{"marker":"1_1591260675.106698_580.9","last_update":"2020-06-04T08:51:15.106698Z"}] 2020-06-04T08:51:56.988 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-06-04T08:51:56.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:56.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:56.990 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:57.041 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:57.041 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:57.300 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":16507416453976119466},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260675.106698_228.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.106698Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591260675.106698_559.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.106698Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591260675.106698_374.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.106698Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260675.106698_580.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.106698Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:51:57.301 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2020-06-04T08:51:57.301 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:57.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:57.302 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:57.341 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:57.341 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:57.610 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260675.287391_197.9","last_update":"2020-06-04T08:51:15.287391Z"},{"marker":"1_1591260675.287391_442.9","last_update":"2020-06-04T08:51:15.287391Z"},{"marker":"1_1591260675.287391_289.6","last_update":"2020-06-04T08:51:15.287391Z"},{"marker":"1_1591260675.287391_359.9","last_update":"2020-06-04T08:51:15.287391Z"}] 2020-06-04T08:51:57.611 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-06-04T08:51:57.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:57.612 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:57.613 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:57.705 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:57.705 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:57.954 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":3830535810264008500},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260675.287391_197.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.287391Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591260675.287391_442.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.287391Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591260675.287391_289.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.287391Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260675.287391_359.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.287391Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:51:57.954 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2020-06-04T08:51:57.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:57.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:57.956 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:58.034 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:58.034 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:58.290 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260675.287391_197.9","last_update":"2020-06-04T08:51:15.287391Z"},{"marker":"1_1591260675.287391_442.9","last_update":"2020-06-04T08:51:15.287391Z"},{"marker":"1_1591260675.287391_289.6","last_update":"2020-06-04T08:51:15.287391Z"},{"marker":"1_1591260675.287391_359.9","last_update":"2020-06-04T08:51:15.287391Z"}] 2020-06-04T08:51:58.291 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-06-04T08:51:58.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:58.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:51:58.293 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:51:58.336 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:51:58.336 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:51:58.584 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9078143951431167771},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1591260675.287391_197.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.287391Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1591260675.287391_442.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.287391Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1591260675.287391_289.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.287391Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1591260675.287391_359.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:51:15.287391Z"}}]},"full_sync":{"total":0,"complete":0}} 2020-06-04T08:51:58.585 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2020-06-04T08:51:58.586 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2020-06-04T08:51:58.587 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-65 2020-06-04T08:51:58.632 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gpxthc-66 2020-06-04T08:51:58.699 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=gpxthc-67 2020-06-04T08:52:00.614 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:00.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:00.616 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:00.659 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:00.660 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:00.943 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000011.85.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:52:00.944 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-65 2020-06-04T08:52:00.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:00.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:00.946 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:00.990 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:00.991 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:01.255 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.5","timestamp":"2020-06-04T08:52:00.404503Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:52:01.256 WARNING:rgw_multi.tests:bucket gpxthc-65 zone test-zone2 behind zone test-zone1: shard=5 master=00000000011.85.5 target=00000000004.79.5 2020-06-04T08:52:06.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:06.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:06.259 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:06.303 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:06.304 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:06.561 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.5","timestamp":"2020-06-04T08:52:00.404503Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:52:06.562 WARNING:rgw_multi.tests:bucket gpxthc-65 zone test-zone2 behind zone test-zone1: shard=5 master=00000000011.85.5 target=00000000004.79.5 2020-06-04T08:52:11.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:11.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:11.565 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:11.610 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:11.610 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:11.860 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.5","timestamp":"2020-06-04T08:52:00.404503Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:52:11.860 WARNING:rgw_multi.tests:bucket gpxthc-65 zone test-zone2 behind zone test-zone1: shard=5 master=00000000011.85.5 target=00000000004.79.5 2020-06-04T08:52:16.861 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:16.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:16.864 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:16.910 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:16.911 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:17.154 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.5","timestamp":"2020-06-04T08:52:00.404503Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:52:17.154 WARNING:rgw_multi.tests:bucket gpxthc-65 zone test-zone2 behind zone test-zone1: shard=5 master=00000000011.85.5 target=00000000004.79.5 2020-06-04T08:52:22.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:22.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:22.159 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:22.204 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:22.204 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:22.468 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.5","timestamp":"2020-06-04T08:52:00.404503Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:52:22.469 WARNING:rgw_multi.tests:bucket gpxthc-65 zone test-zone2 behind zone test-zone1: shard=5 master=00000000011.85.5 target=00000000004.79.5 2020-06-04T08:52:27.471 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:27.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:27.473 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:27.518 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:27.518 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:27.776 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.5","timestamp":"2020-06-04T08:52:00.404503Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:52:27.777 WARNING:rgw_multi.tests:bucket gpxthc-65 zone test-zone2 behind zone test-zone1: shard=5 master=00000000011.85.5 target=00000000004.79.5 2020-06-04T08:52:32.779 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:32.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:32.781 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:32.826 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:32.827 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:33.073 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.85.5","timestamp":"2020-06-04T08:52:00.542008Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:52:33.074 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-65 2020-06-04T08:52:33.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:33.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:33.075 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:33.116 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:33.117 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:33.379 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000011.85.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:52:33.379 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-65 2020-06-04T08:52:33.380 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:33.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:33.382 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:33.465 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:33.465 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:33.735 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.85.5","timestamp":"2020-06-04T08:52:00.542008Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:52:33.736 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-65 2020-06-04T08:52:33.736 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:33.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:33.737 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:33.786 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:33.787 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:34.039 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000011.85.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:52:34.040 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-65 2020-06-04T08:52:34.040 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:34.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:34.041 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:34.088 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:34.089 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:34.351 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.85.5","timestamp":"2020-06-04T08:52:00.542008Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:52:34.351 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-65 2020-06-04T08:52:34.352 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:34.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:34.353 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:34.442 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:34.443 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:34.683 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.60.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:52:34.684 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-65 2020-06-04T08:52:34.684 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:34.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:34.686 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:34.729 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:34.730 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:35.023 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5","timestamp":"2020-06-04T08:52:32.629563Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:52:35.024 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-65 2020-06-04T08:52:35.024 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:35.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:35.025 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:35.065 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:35.065 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:35.301 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.60.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:52:35.301 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-65 2020-06-04T08:52:35.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:35.302 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:35.303 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:35.350 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:35.350 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:35.604 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5","timestamp":"2020-06-04T08:52:32.629617Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:52:35.605 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-65 2020-06-04T08:52:35.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:35.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:35.608 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:35.691 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:35.692 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:35.951 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.73.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:52:35.952 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-65 2020-06-04T08:52:35.952 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:35.953 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:35.954 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:36.037 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:36.037 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:36.303 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5","timestamp":"2020-06-04T08:52:32.813407Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:52:36.304 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-65 2020-06-04T08:52:36.305 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:36.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:36.307 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:36.354 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:36.355 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:36.607 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.73.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:52:36.608 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-65 2020-06-04T08:52:36.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:36.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:36.610 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:36.655 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:36.656 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:36.922 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5","timestamp":"2020-06-04T08:52:32.863607Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:52:36.923 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-65 2020-06-04T08:52:36.924 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-65 zones={test-zone2, test-zone1} 2020-06-04T08:52:36.937 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:52:36.991 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-06-04T08:52:36.991 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:52:37.004 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-06-04T08:52:37.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-06-04T08:52:37.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-06-04T08:52:37.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-06-04T08:52:38.180 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-65 zones={test-zone2, test-zone1} 2020-06-04T08:52:38.181 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-65 zones={test-zone3, test-zone1} 2020-06-04T08:52:38.191 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:52:38.242 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-06-04T08:52:38.243 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:52:38.254 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-06-04T08:52:38.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-06-04T08:52:38.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-06-04T08:52:38.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-06-04T08:52:39.376 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-65 zones={test-zone3, test-zone1} 2020-06-04T08:52:39.377 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-65 zones={test-zone3, test-zone2} 2020-06-04T08:52:39.389 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:52:39.446 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-06-04T08:52:39.447 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:52:39.456 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2020-06-04T08:52:39.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-06-04T08:52:39.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-06-04T08:52:40.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2020-06-04T08:52:40.586 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-65 zones={test-zone3, test-zone2} 2020-06-04T08:52:40.587 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multipart_object_sync ... ok 2020-06-04T08:52:40.588 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-68 2020-06-04T08:52:40.718 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:40.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:40.720 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:40.722 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-06-04T08:52:40.712+0000 7f14447f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:52:40.722 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-06-04T08:52:40.712+0000 7f143d7ea700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:52:40.722 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:52:40.719+0000 7f84df7f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:52:40.723 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:52:40.719+0000 7f84e9ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:52:40.723 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:52:40.719+0000 7f53bcff1700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:52:40.723 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:52:40.719+0000 7f53d4ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:52:40.763 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:40.764 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:40.938 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=387a2dc2-99d4-41c4-ad70-50eff9e4eab1 2020-06-04T08:52:40.966 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260760.624619_342.1","last_update":"2020-06-04T08:52:40.624619Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:52:40.967 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:52:40.968 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:40.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:40.970 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:41.013 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:41.014 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:41.224 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260760.624619_342.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:52:40.624619Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:52:41.225 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:52:41.226 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:52:41.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:41.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:41.228 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:41.271 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:41.271 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:41.522 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260760.624619_342.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:52:40.624619Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:52:41.523 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:52:41.524 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:52:41.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:41.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:41.525 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:41.566 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:41.566 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:41.836 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260760.624619_342.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:52:40.624619Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:52:41.837 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:52:41.838 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:41.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:41.839 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-68 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:41.879 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:41.880 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:42.133 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.62.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.56.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2020-06-04T08:52:42.134 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-68 2020-06-04T08:52:42.135 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-68', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:42.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-68', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:42.136 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-68 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:42.176 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:42.176 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:42.426 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"testobj-sse-c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-04T08:52:40.748761Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"testobj-sse-kms","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-06-04T08:52:40.823444Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}}] 2020-06-04T08:52:42.427 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-68 2020-06-04T08:52:42.530 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_encrypted_object_sync ... ok 2020-06-04T08:52:42.531 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-69 2020-06-04T08:52:42.734 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:52:42.735+0000 7f53c27fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:52:42.734 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:52:42.735+0000 7f53c3fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:52:42.735 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-06-04T08:52:42.728+0000 7f14447f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:52:42.735 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-06-04T08:52:42.728+0000 7f143ffef700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:52:42.735 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-06-04T08:52:42.728+0000 7f143d7ea700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:52:42.737 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:52:42.735+0000 7f84df7f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:52:42.737 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:52:42.735+0000 7f84e37fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:52:42.737 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:52:42.735+0000 7f84e9ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:52:42.737 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:52:42.735+0000 7f53d5ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:52:42.790 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:42.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:42.791 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:42.828 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:42.829 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:43.009 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=387a2dc2-99d4-41c4-ad70-50eff9e4eab1 2020-06-04T08:52:43.041 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260762.570233_367.1","last_update":"2020-06-04T08:52:42.570233Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:52:43.042 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:52:43.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:43.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:43.045 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:43.085 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:43.085 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:43.285 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260762.570233_367.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:52:42.570233Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:52:43.286 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:52:43.287 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:52:43.287 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:43.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:43.288 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:43.325 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:43.325 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:43.557 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260762.570233_367.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:52:42.570233Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:52:43.557 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:52:43.558 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:52:43.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:43.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:43.560 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:43.599 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:43.600 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:43.872 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260762.570233_367.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:52:42.570233Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:52:43.873 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:52:43.874 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:43.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:43.876 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:43.919 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:43.919 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:44.164 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.89.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.62.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.78.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.85.5"}]} 2020-06-04T08:52:44.165 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-69 2020-06-04T08:52:44.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:44.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:44.167 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:44.212 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:44.213 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:44.470 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.89.5","timestamp":"2020-06-04T08:52:42.863508Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-04T08:52:43.027508Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.78.5","timestamp":"2020-06-04T08:52:42.871435Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5","timestamp":"2020-06-04T08:52:42.863543Z"}}}] 2020-06-04T08:52:44.471 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-69 2020-06-04T08:52:44.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:44.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:44.473 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:44.518 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:44.518 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:44.770 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.89.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.62.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.78.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.85.5"}]} 2020-06-04T08:52:44.771 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-69 2020-06-04T08:52:44.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:44.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:44.773 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:44.856 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:44.857 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:45.132 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.89.5","timestamp":"2020-06-04T08:52:42.845497Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-04T08:52:43.021408Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.78.5","timestamp":"2020-06-04T08:52:42.849426Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5","timestamp":"2020-06-04T08:52:42.845533Z"}}}] 2020-06-04T08:52:45.133 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-69 2020-06-04T08:52:45.133 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:45.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:45.135 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:45.215 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:45.215 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:45.478 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.89.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.62.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.78.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.85.5"}]} 2020-06-04T08:52:45.479 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-69 2020-06-04T08:52:45.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:45.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:45.481 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:45.524 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:45.525 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:45.800 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.89.5","timestamp":"2020-06-04T08:52:42.581638Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.62.5","timestamp":"2020-06-04T08:52:42.740158Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.78.5","timestamp":"2020-06-04T08:52:42.692248Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.85.5","timestamp":"2020-06-04T08:52:42.637443Z"}}}] 2020-06-04T08:52:45.801 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-69 2020-06-04T08:52:45.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:45.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:45.808 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:45.883 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:45.883 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:46.130 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.70.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.56.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.107.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.118.5"}]} 2020-06-04T08:52:46.131 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-69 2020-06-04T08:52:46.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:46.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:46.134 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:46.179 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:46.179 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:46.458 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.70.5","timestamp":"2020-06-04T08:52:43.033478Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-06-04T08:52:43.277462Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.107.5","timestamp":"2020-06-04T08:52:43.041427Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.118.5","timestamp":"2020-06-04T08:52:43.033507Z"}}}] 2020-06-04T08:52:46.459 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-69 2020-06-04T08:52:46.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:46.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:46.461 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:46.504 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:46.504 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:46.747 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.70.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.56.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.107.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.118.5"}]} 2020-06-04T08:52:46.747 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-69 2020-06-04T08:52:46.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:46.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:46.755 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:46.807 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:46.808 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:47.056 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.70.5","timestamp":"2020-06-04T08:52:43.033556Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.56.5","timestamp":"2020-06-04T08:52:43.277462Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.107.5","timestamp":"2020-06-04T08:52:43.033513Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.118.5","timestamp":"2020-06-04T08:52:43.033542Z"}}}] 2020-06-04T08:52:47.057 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-69 2020-06-04T08:52:47.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:47.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:47.058 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:47.103 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:47.104 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:47.347 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.83.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.65.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.78.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.111.5"}]} 2020-06-04T08:52:47.348 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-69 2020-06-04T08:52:47.348 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:47.349 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:47.350 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:47.433 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:47.433 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:47.719 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.83.5","timestamp":"2020-06-04T08:52:43.269442Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.65.5","timestamp":"2020-06-04T08:52:43.265459Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.78.5","timestamp":"2020-06-04T08:52:43.017407Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.111.5","timestamp":"2020-06-04T08:52:43.213910Z"}}}] 2020-06-04T08:52:47.734 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-69 2020-06-04T08:52:47.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:47.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:47.736 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:47.778 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:47.778 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:48.018 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.83.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.65.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.78.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.111.5"}]} 2020-06-04T08:52:48.018 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-69 2020-06-04T08:52:48.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:48.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:48.020 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:48.069 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:48.069 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:48.333 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.83.5","timestamp":"2020-06-04T08:52:43.267548Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.65.5","timestamp":"2020-06-04T08:52:43.267519Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.78.5","timestamp":"2020-06-04T08:52:43.027512Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.111.5","timestamp":"2020-06-04T08:52:43.219540Z"}}}] 2020-06-04T08:52:48.334 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-69 2020-06-04T08:52:48.334 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-69 zones={test-zone2, test-zone1} 2020-06-04T08:52:48.347 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:52:48.354 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-04T08:52:48.354 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-04T08:52:48.355 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-04T08:52:48.355 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-04T08:52:48.356 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:52:48.364 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-04T08:52:48.364 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-04T08:52:48.365 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-04T08:52:48.365 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-04T08:52:48.380 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-04T08:52:48.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-04T08:52:48.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-04T08:52:48.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-04T08:52:48.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-04T08:52:48.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-04T08:52:48.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-04T08:52:48.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-04T08:52:49.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-04T08:52:49.224 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-04T08:52:49.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-04T08:52:49.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-04T08:52:49.516 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-69 zones={test-zone2, test-zone1} 2020-06-04T08:52:49.516 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-69 zones={test-zone3, test-zone1} 2020-06-04T08:52:49.525 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:52:49.531 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-04T08:52:49.532 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-04T08:52:49.532 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-04T08:52:49.532 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-04T08:52:49.533 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:52:49.539 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-04T08:52:49.539 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-04T08:52:49.540 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-04T08:52:49.540 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-04T08:52:49.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-04T08:52:49.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-04T08:52:49.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-04T08:52:49.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-04T08:52:49.916 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-04T08:52:50.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-04T08:52:50.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-04T08:52:50.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-04T08:52:50.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-04T08:52:50.462 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-04T08:52:50.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-04T08:52:50.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-04T08:52:50.752 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-69 zones={test-zone3, test-zone1} 2020-06-04T08:52:50.753 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-69 zones={test-zone3, test-zone2} 2020-06-04T08:52:50.763 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:52:50.771 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-04T08:52:50.772 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-04T08:52:50.772 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-04T08:52:50.773 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-04T08:52:50.773 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:52:50.781 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-04T08:52:50.782 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-04T08:52:50.782 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-04T08:52:50.782 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-04T08:52:50.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-04T08:52:50.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-04T08:52:50.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-04T08:52:51.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-04T08:52:51.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-04T08:52:51.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-04T08:52:51.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-04T08:52:51.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-04T08:52:51.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-04T08:52:51.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-04T08:52:51.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-04T08:52:51.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-04T08:52:51.937 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-69 zones={test-zone3, test-zone2} 2020-06-04T08:52:51.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:52:51.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:52:51.950 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:52:51.989 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:51.989 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:52.261 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gpxthc-70 2020-06-04T08:52:52.337 INFO:tasks.rgw.c2.client.0.smithi196.stdout:2020-06-04T08:52:52.334+0000 7f84ea7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:52:52.338 INFO:tasks.rgw.c1.client.1.smithi060.stdout:2020-06-04T08:52:52.332+0000 7f145a7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:52:52.339 INFO:tasks.rgw.c2.client.1.smithi196.stdout:2020-06-04T08:52:52.338+0000 7f53bdff3700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2020-06-04T08:52:52.510 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:52.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:52.511 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:52.551 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:52.552 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:52.727 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=387a2dc2-99d4-41c4-ad70-50eff9e4eab1 2020-06-04T08:52:52.759 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260772.293207_372.1","last_update":"2020-06-04T08:52:52.293207Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:52:52.766 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:52:52.767 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:52.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:52.768 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:52.819 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:52.819 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:53.036 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260772.293207_372.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:52:52.293207Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:52:53.037 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:52:53.038 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:52:53.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:53.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:53.040 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:53.083 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:53.084 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:53.322 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260772.293207_372.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:52:52.293207Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:52:53.323 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:52:53.323 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:52:53.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:53.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:53.325 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:53.367 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:53.368 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:53.644 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260772.293207_372.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:52:52.293207Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:52:53.644 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:52:53.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:53.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:53.648 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:53.689 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:53.690 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:53.949 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.112.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.41.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.78.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.203.5"}]} 2020-06-04T08:52:53.949 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-70 2020-06-04T08:52:53.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:53.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:53.952 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:54.004 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:54.005 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:54.243 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.112.5","timestamp":"2020-06-04T08:52:52.451624Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5","timestamp":"2020-06-04T08:52:52.539463Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.78.5","timestamp":"2020-06-04T08:52:52.491377Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.203.5","timestamp":"2020-06-04T08:52:52.495517Z"}}}] 2020-06-04T08:52:54.244 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gpxthc-70 2020-06-04T08:52:54.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:54.246 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:54.247 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:54.289 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:54.289 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:54.547 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.112.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.41.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.78.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.203.5"}]} 2020-06-04T08:52:54.547 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-70 2020-06-04T08:52:54.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:54.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:54.549 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:54.633 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:54.633 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:54.911 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.112.5","timestamp":"2020-06-04T08:52:52.441462Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5","timestamp":"2020-06-04T08:52:52.525449Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.78.5","timestamp":"2020-06-04T08:52:52.541109Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.203.5","timestamp":"2020-06-04T08:52:52.625356Z"}}}] 2020-06-04T08:52:54.911 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gpxthc-70 2020-06-04T08:52:54.912 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:54.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:54.913 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:54.957 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:54.957 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:55.209 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.112.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.41.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.78.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.203.5"}]} 2020-06-04T08:52:55.210 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-70 2020-06-04T08:52:55.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:55.211 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:55.214 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:55.260 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:55.261 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:55.529 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.112.5","timestamp":"2020-06-04T08:52:52.303895Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.41.5","timestamp":"2020-06-04T08:52:52.465032Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.78.5","timestamp":"2020-06-04T08:52:52.413142Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.203.5","timestamp":"2020-06-04T08:52:52.367938Z"}}}] 2020-06-04T08:52:55.530 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-70 2020-06-04T08:52:55.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:55.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:55.531 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:55.619 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:55.620 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:55.860 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.108.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.67.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.75.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.198.5"}]} 2020-06-04T08:52:55.860 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-70 2020-06-04T08:52:55.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:55.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:55.863 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:55.909 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:55.910 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:56.192 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.108.5","timestamp":"2020-06-04T08:52:52.633440Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5","timestamp":"2020-06-04T08:52:52.841602Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.75.5","timestamp":"2020-06-04T08:52:52.637415Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.198.5","timestamp":"2020-06-04T08:52:52.845401Z"}}}] 2020-06-04T08:52:56.193 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gpxthc-70 2020-06-04T08:52:56.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:56.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:56.196 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:56.240 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:56.241 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:56.486 INFO:teuthology.orchestra.run.smithi196.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.108.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.67.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.75.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.198.5"}]} 2020-06-04T08:52:56.486 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-70 2020-06-04T08:52:56.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:56.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gpxthc-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:56.488 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gpxthc-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:56.535 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:56.535 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:56.787 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.108.5","timestamp":"2020-06-04T08:52:52.677422Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5","timestamp":"2020-06-04T08:52:52.837663Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.75.5","timestamp":"2020-06-04T08:52:52.637401Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.198.5","timestamp":"2020-06-04T08:52:52.837511Z"}}}] 2020-06-04T08:52:56.788 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=gpxthc-70 2020-06-04T08:52:56.789 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:56.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:56.790 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:56.829 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:56.829 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:57.089 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.87.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.51.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.76.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.210.5"}]} 2020-06-04T08:52:57.091 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-70 2020-06-04T08:52:57.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:57.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:57.093 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:57.176 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:57.177 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:57.468 INFO:teuthology.orchestra.run.smithi060.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.87.5","timestamp":"2020-06-04T08:52:52.617401Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5","timestamp":"2020-06-04T08:52:52.853358Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5","timestamp":"2020-06-04T08:52:52.825402Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.210.5","timestamp":"2020-06-04T08:52:53.061564Z"}}}] 2020-06-04T08:52:57.469 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=gpxthc-70 2020-06-04T08:52:57.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:57.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:57.470 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:57.550 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:57.551 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:57.792 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.87.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.51.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.76.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.210.5"}]} 2020-06-04T08:52:57.792 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-70 2020-06-04T08:52:57.793 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:57.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'gpxthc-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:52:57.794 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket gpxthc-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:52:57.840 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:52:57.841 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:52:58.097 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.87.5","timestamp":"2020-06-04T08:52:52.667422Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5","timestamp":"2020-06-04T08:52:52.867404Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5","timestamp":"2020-06-04T08:52:52.807467Z"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.210.5","timestamp":"2020-06-04T08:52:53.067614Z"}}}] 2020-06-04T08:52:58.097 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=gpxthc-70 2020-06-04T08:52:58.098 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-70 zones={test-zone2, test-zone1} 2020-06-04T08:52:58.109 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:52:58.116 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-04T08:52:58.116 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-04T08:52:58.117 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-04T08:52:58.117 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-04T08:52:58.118 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:52:58.127 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-04T08:52:58.128 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-04T08:52:58.128 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-04T08:52:58.129 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-04T08:52:58.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-04T08:52:58.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-04T08:52:58.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-04T08:52:58.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-04T08:52:58.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-04T08:52:58.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-04T08:52:58.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-04T08:52:58.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-04T08:52:58.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-04T08:52:59.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-04T08:52:59.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-04T08:52:59.181 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-04T08:52:59.290 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-70 zones={test-zone2, test-zone1} 2020-06-04T08:52:59.291 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-70 zones={test-zone3, test-zone1} 2020-06-04T08:52:59.300 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:52:59.307 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-04T08:52:59.307 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-04T08:52:59.308 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-04T08:52:59.308 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-04T08:52:59.309 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:52:59.316 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-04T08:52:59.316 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-04T08:52:59.317 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-04T08:52:59.317 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-04T08:52:59.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-04T08:52:59.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-04T08:52:59.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-04T08:52:59.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-04T08:52:59.670 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-04T08:52:59.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-04T08:52:59.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-04T08:53:00.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-04T08:53:00.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-04T08:53:00.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-04T08:53:00.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-04T08:53:00.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-04T08:53:00.557 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-70 zones={test-zone3, test-zone1} 2020-06-04T08:53:00.559 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gpxthc-70 zones={test-zone3, test-zone2} 2020-06-04T08:53:00.570 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2020-06-04T08:53:00.578 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-04T08:53:00.579 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-04T08:53:00.580 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-04T08:53:00.581 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-04T08:53:00.581 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2020-06-04T08:53:00.590 DEBUG:tasks.rgw_multi.zone_rados:o=a 2020-06-04T08:53:00.591 DEBUG:tasks.rgw_multi.zone_rados:o=b 2020-06-04T08:53:00.592 DEBUG:tasks.rgw_multi.zone_rados:o=c 2020-06-04T08:53:00.593 DEBUG:tasks.rgw_multi.zone_rados:o=d 2020-06-04T08:53:00.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-04T08:53:00.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-04T08:53:00.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2020-06-04T08:53:00.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-04T08:53:00.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-04T08:53:01.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2020-06-04T08:53:01.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-04T08:53:01.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-04T08:53:01.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2020-06-04T08:53:01.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-04T08:53:01.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-04T08:53:01.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2020-06-04T08:53:01.794 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gpxthc-70 zones={test-zone3, test-zone2} 2020-06-04T08:53:01.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:53:01.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:53:01.796 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:53:01.832 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:01.832 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:02.105 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'gpxthc-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:02.106 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'gpxthc-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:02.106 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket gpxthc-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:02.186 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:02.187 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:02.445 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2020-06-04T08:53:02.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'gpxthc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:02.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'gpxthc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:02.448 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket gpxthc-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:02.525 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:02.525 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:02.778 INFO:teuthology.orchestra.run.smithi060.stdout:[{"op_id":"1#00000000001.89.5","op_tag":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7713.4529","op":"write","object":"a","instance":"","state":"complete","index_ver":1,"timestamp":"2020-06-04T08:52:42.581638558Z","ver":{"pool":9,"epoch":2},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-69:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.20"}]},{"op_id":"5#00000000001.62.5","op_tag":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7713.4562","op":"write","object":"d","instance":"","state":"complete","index_ver":1,"timestamp":"2020-06-04T08:52:42.740158494Z","ver":{"pool":9,"epoch":1},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-69:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.20"}]},{"op_id":"8#00000000001.78.5","op_tag":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7713.4533","op":"write","object":"c","instance":"","state":"complete","index_ver":1,"timestamp":"2020-06-04T08:52:42.692248670Z","ver":{"pool":9,"epoch":5},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-69:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.20"}]},{"op_id":"10#00000000001.85.5","op_tag":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7713.4531","op":"write","object":"b","instance":"","state":"complete","index_ver":1,"timestamp":"2020-06-04T08:52:42.637443796Z","ver":{"pool":9,"epoch":4},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"7ad4cc51-2d19-48f1-a21e-ef15faf62b94:gpxthc-69:7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.20"}]}] 2020-06-04T08:53:02.779 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:53:02.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2020-06-04T08:53:02.780 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2020-06-04T08:53:02.853 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:02.854 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:04.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'gpxthc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:04.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'gpxthc-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:04.518 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket gpxthc-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:04.607 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:04.608 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:04.853 INFO:teuthology.orchestra.run.smithi060.stdout:[] 2020-06-04T08:53:04.854 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_index_log_trim ... ok 2020-06-04T08:53:04.900 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_creation_time ... ok 2020-06-04T08:53:04.901 INFO:tasks.rgw_multisite_tests: 2020-06-04T08:53:04.902 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2020-06-04T08:53:04.902 INFO:tasks.rgw_multisite_tests:Ran 28 tests in 1447.031s 2020-06-04T08:53:04.903 INFO:tasks.rgw_multisite_tests: 2020-06-04T08:53:04.903 INFO:tasks.rgw_multisite_tests:OK 2020-06-04T08:53:04.907 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2020-06-04T08:53:04.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:04.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:04.910 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:04.950 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:04.951 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:05.171 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=387a2dc2-99d4-41c4-ad70-50eff9e4eab1 2020-06-04T08:53:05.210 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260772.293207_372.1","last_update":"2020-06-04T08:52:52.293207Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:53:05.211 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:53:05.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:05.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:05.212 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:05.252 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:05.252 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:05.488 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260772.293207_372.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:52:52.293207Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:53:05.489 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:53:05.490 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:53:05.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:05.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:05.491 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:05.535 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:05.535 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:05.769 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260772.293207_372.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:52:52.293207Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:53:05.770 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:53:05.770 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:53:05.771 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:05.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:05.773 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:05.815 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:05.815 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:06.095 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260772.293207_372.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:52:52.293207Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:53:06.095 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:53:06.097 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:06.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:06.099 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:06.141 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:06.142 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:06.360 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=387a2dc2-99d4-41c4-ad70-50eff9e4eab1 2020-06-04T08:53:06.399 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260772.293207_372.1","last_update":"2020-06-04T08:52:52.293207Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:53:06.399 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:53:06.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:06.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:06.401 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:06.443 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:06.444 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:06.715 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260772.293207_372.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:52:52.293207Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:53:06.716 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:53:06.762 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:06.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:06.763 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:06.798 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:06.798 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:06.996 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=387a2dc2-99d4-41c4-ad70-50eff9e4eab1 2020-06-04T08:53:07.026 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260786.754557_381.1","last_update":"2020-06-04T08:53:06.754557Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:53:07.027 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:53:07.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:07.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:07.028 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:07.066 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:07.066 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:07.327 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260786.754557_381.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:53:06.754557Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:53:07.328 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:53:07.457 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:07.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:07.459 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:07.494 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:07.494 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:07.722 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=387a2dc2-99d4-41c4-ad70-50eff9e4eab1 2020-06-04T08:53:07.762 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260786.754557_381.1","last_update":"2020-06-04T08:53:06.754557Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:53:07.763 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:53:07.763 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:07.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:07.764 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:07.804 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:07.805 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:08.079 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260786.754557_381.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:53:06.754557Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:53:08.079 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:53:08.247 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... ok 2020-06-04T08:53:08.248 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:08.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:08.249 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:08.291 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:08.292 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:08.511 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=387a2dc2-99d4-41c4-ad70-50eff9e4eab1 2020-06-04T08:53:08.545 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260788.236253_383.1","last_update":"2020-06-04T08:53:08.236253Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:53:08.546 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:53:08.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:08.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:08.548 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:08.589 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:08.590 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:08.837 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260788.236253_383.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:53:08.236253Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:53:08.837 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:53:08.838 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:53:08.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:08.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:08.840 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:08.880 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:08.881 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:09.124 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260788.236253_383.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:53:08.236253Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:53:09.125 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:53:09.125 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:53:09.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:09.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:09.127 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:09.165 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:09.165 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:09.427 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260788.236253_383.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:53:08.236253Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:53:09.428 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:53:09.429 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:09.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:09.431 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:09.470 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:09.688 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=387a2dc2-99d4-41c4-ad70-50eff9e4eab1 2020-06-04T08:53:09.736 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260788.236253_383.1","last_update":"2020-06-04T08:53:08.236253Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:53:09.737 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:53:09.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:09.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:09.738 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:09.780 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:09.780 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:10.054 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260788.236253_383.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:53:08.236253Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:53:10.055 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:53:10.107 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:10.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:10.109 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:10.151 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:10.152 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:10.354 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=387a2dc2-99d4-41c4-ad70-50eff9e4eab1 2020-06-04T08:53:10.394 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260790.097040_387.1","last_update":"2020-06-04T08:53:10.097040Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:53:10.395 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:53:10.396 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:10.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:10.397 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:10.438 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:10.439 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:10.719 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260790.097040_387.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:53:10.097040Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:53:10.720 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:53:10.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:10.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:10.794 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:10.835 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:10.835 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:11.060 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=387a2dc2-99d4-41c4-ad70-50eff9e4eab1 2020-06-04T08:53:11.098 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260790.097040_387.1","last_update":"2020-06-04T08:53:10.097040Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:53:11.099 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:53:11.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:11.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:11.102 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:11.142 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:11.143 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:11.407 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260790.097040_387.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:53:10.097040Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:53:11.408 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:53:11.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:11.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gpxthc-72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:11.848 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gpxthc-72 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:11.903 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:11.903 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:12.151 INFO:teuthology.orchestra.run.smithi060.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.101.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.67.5"},{"key":4,"val":"00000000001.82.5"},{"key":5,"val":"00000000001.97.5"},{"key":6,"val":""},{"key":7,"val":"00000000001.83.5"},{"key":8,"val":"00000000001.120.5"},{"key":9,"val":"00000000001.106.5"},{"key":10,"val":"00000000002.232.5"}]} 2020-06-04T08:53:12.152 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-72 2020-06-04T08:53:12.152 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-72', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:12.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gpxthc-72', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:12.154 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gpxthc-72 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:12.198 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:12.198 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:12.459 INFO:teuthology.orchestra.run.smithi196.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.101.5","timestamp":"2020-06-04T08:53:11.749671Z"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.67.5","timestamp":"2020-06-04T08:53:11.412143Z"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.82.5","timestamp":"2020-06-04T08:53:11.706243Z"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.97.5","timestamp":"2020-06-04T08:53:11.522572Z"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"","timestamp":"0.000000"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.83.5","timestamp":"2020-06-04T08:53:11.478150Z"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.120.5","timestamp":"2020-06-04T08:53:11.662313Z"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.106.5","timestamp":"2020-06-04T08:53:11.432252Z"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.232.5","timestamp":"2020-06-04T08:53:11.797229Z"}}}] 2020-06-04T08:53:12.459 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=gpxthc-72 2020-06-04T08:53:12.470 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-06-04T08:53:11.522572Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'gpxthc-72_notif', 'bucket': {'name': 'gpxthc-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::gpxthc-72', 'id': '7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.23'}, 'object': {'key': '3', 'size': 0, 'etag': '70ec95fa', 'versionId': '', 'sequencer': '77B6D85E082CA224', 'metadata': [], 'tags': []}}, 'eventId': '1591260791.614607.70ec95fa', 'opaqueData': ''} 2020-06-04T08:53:12.471 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-06-04T08:53:11.412143Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'gpxthc-72_notif', 'bucket': {'name': 'gpxthc-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::gpxthc-72', 'id': '7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.23'}, 'object': {'key': '0', 'size': 0, 'etag': '59225ad0', 'versionId': '', 'sequencer': '77B6D85ECCE9D524', 'metadata': [], 'tags': []}}, 'eventId': '1591260791.617998.59225ad0', 'opaqueData': ''} 2020-06-04T08:53:12.471 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-06-04T08:53:11.478150Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'gpxthc-72_notif', 'bucket': {'name': 'gpxthc-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::gpxthc-72', 'id': '7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.23'}, 'object': {'key': '2', 'size': 0, 'etag': '1f965b85', 'versionId': '', 'sequencer': '77B6D85EAEEC7B27', 'metadata': [], 'tags': []}}, 'eventId': '1591260791.662432.1f965b85', 'opaqueData': ''} 2020-06-04T08:53:12.472 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-06-04T08:53:11.432252Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'gpxthc-72_notif', 'bucket': {'name': 'gpxthc-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::gpxthc-72', 'id': '7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.23'}, 'object': {'key': '1', 'size': 0, 'etag': 'd90c82f3', 'versionId': '', 'sequencer': '77B6D85E483C2F2A', 'metadata': [], 'tags': []}}, 'eventId': '1591260791.707738.d90c82f3', 'opaqueData': ''} 2020-06-04T08:53:12.472 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-06-04T08:53:11.706243Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'gpxthc-72_notif', 'bucket': {'name': 'gpxthc-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::gpxthc-72', 'id': '7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.23'}, 'object': {'key': '7', 'size': 0, 'etag': 'a27478ed', 'versionId': '', 'sequencer': '77B6D85E55668E30', 'metadata': [], 'tags': []}}, 'eventId': '1591260791.814638.a27478ed', 'opaqueData': ''} 2020-06-04T08:53:12.473 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-06-04T08:53:11.565641Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'gpxthc-72_notif', 'bucket': {'name': 'gpxthc-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::gpxthc-72', 'id': '7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.23'}, 'object': {'key': '4', 'size': 0, 'etag': '0e33ab92', 'versionId': '', 'sequencer': '77B6D85EA239DB32', 'metadata': [], 'tags': []}}, 'eventId': '1591260791.853227.0e33ab92', 'opaqueData': ''} 2020-06-04T08:53:12.473 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-06-04T08:53:11.749671Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'gpxthc-72_notif', 'bucket': {'name': 'gpxthc-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::gpxthc-72', 'id': '7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.23'}, 'object': {'key': '8', 'size': 0, 'etag': '3176a79a', 'versionId': '', 'sequencer': '77B6D85E52206535', 'metadata': [], 'tags': []}}, 'eventId': '1591260791.895819.3176a79a', 'opaqueData': ''} 2020-06-04T08:53:12.474 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-06-04T08:53:11.662313Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'gpxthc-72_notif', 'bucket': {'name': 'gpxthc-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::gpxthc-72', 'id': '7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.23'}, 'object': {'key': '6', 'size': 0, 'etag': 'ec1eff4a', 'versionId': '', 'sequencer': '77B6D85E81E7E335', 'metadata': [], 'tags': []}}, 'eventId': '1591260791.904128.ec1eff4a', 'opaqueData': ''} 2020-06-04T08:53:12.474 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-06-04T08:53:11.797229Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'gpxthc-72_notif', 'bucket': {'name': 'gpxthc-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::gpxthc-72', 'id': '7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.23'}, 'object': {'key': '9', 'size': 0, 'etag': '9aaa9264', 'versionId': '', 'sequencer': '77B6D85ED22A7D38', 'metadata': [], 'tags': []}}, 'eventId': '1591260791.947727.9aaa9264', 'opaqueData': ''} 2020-06-04T08:53:12.475 DEBUG:tasks.rgw_multi.tests_ps:{'eventVersion': '2.2', 'eventSource': 'ceph:s3', 'awsRegion': '', 'eventTime': '2020-06-04T08:53:11.613288Z', 'eventName': 's3:ObjectCreated:*', 'userIdentity': {'principalId': ''}, 'requestParameters': {'sourceIPAddress': ''}, 'responseElements': {'x-amz-request-id': '', 'x-amz-id-2': ''}, 's3': {'s3SchemaVersion': '1.0', 'configurationId': 'gpxthc-72_notif', 'bucket': {'name': 'gpxthc-72', 'ownerIdentity': {'principalId': 'rgw-multisite-test-user'}, 'arn': 'arn:aws:s3:::gpxthc-72', 'id': '7ad4cc51-2d19-48f1-a21e-ef15faf62b94.7721.23'}, 'object': {'key': '5', 'size': 0, 'etag': '89c3ac73', 'versionId': '', 'sequencer': '77B6D85EA73BCA38', 'metadata': [], 'tags': []}}, 'eventId': '1591260791.952777.89c3ac73', 'opaqueData': ''} 2020-06-04T08:53:12.485 WARNING:tasks.rgw_multi.tests_ps:superfluous records are found 2020-06-04T08:53:12.674 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... ok 2020-06-04T08:53:12.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:12.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:12.677 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:12.718 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:12.718 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:12.919 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=387a2dc2-99d4-41c4-ad70-50eff9e4eab1 2020-06-04T08:53:12.956 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260792.663513_393.1","last_update":"2020-06-04T08:53:12.663513Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:53:12.966 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2020-06-04T08:53:12.967 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:12.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:12.969 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:13.015 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:13.016 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:13.257 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260792.663513_393.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:53:12.663513Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:53:13.257 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2020-06-04T08:53:13.258 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2020-06-04T08:53:13.258 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:13.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:13.260 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:13.302 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:13.302 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:13.552 INFO:teuthology.orchestra.run.smithi060.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260792.663513_393.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:53:12.663513Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:53:13.553 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2020-06-04T08:53:13.554 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:53:13.554 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:13.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:13.555 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:13.597 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:13.597 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:13.870 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260792.663513_393.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:53:12.663513Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:53:13.870 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:53:13.871 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:13.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:13.872 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:13.913 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:13.913 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:14.139 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=387a2dc2-99d4-41c4-ad70-50eff9e4eab1 2020-06-04T08:53:14.177 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260792.663513_393.1","last_update":"2020-06-04T08:53:12.663513Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:53:14.178 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:53:14.178 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:14.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:14.180 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:14.218 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:14.219 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:14.496 INFO:teuthology.orchestra.run.smithi196.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"387a2dc2-99d4-41c4-ad70-50eff9e4eab1","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1591260792.663513_393.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2020-06-04T08:53:12.663513Z","realm_epoch":4}},{"key":1,"val":{"state":1,"marker":"1_1591259350.985454_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2020-06-04T08:29:10.985454Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2020-06-04T08:53:14.497 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2020-06-04T08:53:14.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:14.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:14.549 INFO:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:14.586 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:14.586 INFO:teuthology.orchestra.run.smithi060.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T08:53:14.801 INFO:teuthology.orchestra.run.smithi060.stderr:No --period given, using current period=387a2dc2-99d4-41c4-ad70-50eff9e4eab1 2020-06-04T08:53:14.826 INFO:teuthology.orchestra.run.smithi060.stdout:[{"marker":"1_1591260794.539337_397.1","last_update":"2020-06-04T08:53:14.539337Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2020-06-04T08:53:14.826 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2020-06-04T08:53:14.827 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:14.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2020-06-04T08:53:14.828 INFO:teuthology.orchestra.run.smithi196:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2020-06-04T08:53:14.865 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setuser ceph since I am not root 2020-06-04T08:53:14.865 INFO:teuthology.orchestra.run.smithi196.stderr:ignoring --setgroup ceph since I am not root 2020-06-04T20:09:17.185 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-06-04T20:09:17.218 DEBUG:teuthology.task.console_log:Killing console logger for smithi060 2020-06-04T20:09:17.229 DEBUG:teuthology.task.console_log:Killing console logger for smithi196 2020-06-04T20:09:17.229 DEBUG:teuthology.task.console_log:Killing console logger for smithi060 2020-06-04T20:09:17.230 DEBUG:teuthology.task.console_log:Killing console logger for smithi196 2020-06-04T20:09:17.235 DEBUG:teuthology.exit:Finished running handlers